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Because of the pragaatic need of school districts to 
respond to questions posed by those interested in their efforts to 
iapleaent career education and the equally practical need to further 
its definition, there exists a deaand to assess the results of 
efforts to iapleaent the concept. The aain body of this sonograph 
provides an operational approach to aeeting these two needs. It 
outlines steps which a local school district can follow in conducting 
an evaluation of its career education effort. A longer tern result of 
following these steps would be the accuaulation of inforaation 
necessary to deteraine the extent to which the objectives associated 
with the concept have been achieved; that is, the baseline data for 
deteraining whether or not these are individual attributes which 
should be fostered. If the fraaework provided is followed and the 
results are shared with OSOB, it is expected that both local needs 
for assessaent and national needs for policy inforaation and further 
theoretical developaent will be aet. (Author/AJ) 



POB DATE 
COtlTPAC* 
HOT? 

EDRS PPIC? 
DESCPIPTOPS 



ABSTRACT 



V 



EOUC^TlOM 




HANDBOOK FOR THE EVALUATION 
OF CAREER EDUCATION PROGRAMS 
(DRAFT) 



BUREAU OF OCCUPATIONAL AND 
ADULT EDUCATION 
U.S. OFFICE OF EDUCATION 
DEPARTMENT OF HEALTH. EDUCATION, AND WELFARE 
Washington, D.C. 
Under Contract No. OEC-0-73-6663 



Submitted by: 



0 
0 



DEVELOPMENT ASSOCIATES, INC. 
1521 New Hampshire Avenue, N.W. 
Washington, D.C. 20036 



August 15, 197^ 



ERIC 



civil Rights Coi^l lance 



Discrimination Prohlbltod - Title VI of the Civil Rights 
Act of 1964 states: 

'*fto person in the United States shall, on the 
ground of race, color, or national origin, be 
excluded from participation In, be denied the 
benefits of, or be subjected to discrimination 
under any prcxiram or activity receiving Federal 
financial assistance.'* 

Therefore, any progran or activity receiving financial 
assistance from the Department of Health, Education, and 
Welfare must be operated In compliance with this law. 



ERIC 




TABLE OF CONTENTS 

Pago 

CHAPTER I - INTRODl'CTION 1 

An Objectives-Based Definition of Career Edacation , 3 

Assessing Career Education ^ 

CHAPTER II - PLAN FOR EVALUATING CAREER EDUCATION. . 11 

A. The Evaluation Plan 12 

B. Design Work Sheets: An Aid to Planning \^ 

CHAPTER 111 - STEP I: PREPARING AN OVERVIEW CI THE 

PROGRAM 21 

CHAPTER IV - COMPLETION OF OUTCOME QUESTION/ 

TREATMENT GROUP MATRIX 27 

A. Specify Treatment 

B. Identify Students by Treatnnent 33 

C. Identify Treatment Groups 3 5 

D. Complete the Treatment Group - Outcome Area Table 37 

E. Complete the Outcome Question/Treatment Group Matrix ... 40 

CHAPTER V - FORMULATING THE EVALUATION QUESTIONS 

AND SPECIFYING THE EVALUATION DESIGN ... 41 

CHAPTER VI - MEASURING OUTCOME VARIABLES SPECIFYING 

DATA SOURCES AND INSTRUMENTS 55 

A. Dati Sources 55 

B. Instrumentation ^1 

CHAPTER VII - MEASURING PROGRAM TREATMENTS 77 

A. Specifying Data Sources 78 

B. Instrumentation 80 

CHAPTER VIII - SAMPLING PLAN ^-5 

CHAPTER IX . DATA ANALYSIS PLAN "^"^ 

Specifying the Data Analysis Techniques 

References . ^ ' ' 



ERIC 



Page 

Page I — ^ 

CHAPTER X - DATA COLLECTION PLAN ii7 

CHAPTER XI - FINALIZE THE DESIGN AND PREPARE A 

REPORTING PLAN 125 

A. Dpsiijn Review 125 

B. Rt'portint; Plan j^27 

CHAPTER Xn - IMPLEMENTING THE EVALUATION 131 

A. Data Collection , 132^ 

B. Data Processing and Analysis j^^^ 

C. Reporting Findings 135 

APPENDIX 13S* 

A. Acknowledpments I39 

B. Partial Bibliography 141 

C. Response Regarding the Self Observation Scales I43 

D. Response Regarding the Differential Aptitude Test. ....... 145 



ERIC 



LIST OF TABLES 



Ch;*oter 1 



Tabic I ; Career Ediaaiion Objectives in Ti^rms of Student 

Outcome Statenu^nt {, 

Chapter II 

Tabh; II- A; Evaluation Design Worksheet. 20 

Chapter IV 

Table IV-A; Activity- Treatment Table 30 

Table IV>B; Illustration of Activity Treatment Tabic 31 

Table IV- Cr Activity-Treatments- Participants Table 34 

Table IV- D; Participant- Treatment Group Table 36 

Table IV-E; Treatment Group-Outcome Area Table Illustration 38 
Table IV-F: Treatment Group-Outcome Area Table Basic 

Skill 39 

Table IV-G; Illustration of Outcome /Quest ion Treatment Group 

MatrLx , 41 

Table IV- H: Out«.ome/Question Treatment Group Matrix 41 

Ctiapte ' VI 

Tabh^ V"I-A: Instruments Recommended oy Panel 64 

Table VI- B; Outcome Areas for Which no Tests Were 

Recommended , ♦ 67 

Tabic VI- C: Instrimient List h9 

Table VI>D; Tests ilavinj; Promising Approaches btit Insufficient 

Dcvolopnent or Lack of Psychometric Data 72 

Chapter X 

Tabic X-A: Data Collection Plan • , 119 

Table X-B ; Data Collection Plannintf Chart • 120 

Table X-C: Training Needs Assesfment 121 

T.ibli' X-D: Pi^rsonnel Planning Fo -m 122 

Append 

Appendix Table I): \\<}dvl for Group Summary of Data Available 

Tor \\u' Diffi-rcnt ial Aptit'idc J\'ht.. 142 



7 



LIST OF FIGURES 



i: lapter II 

Figure U'A: Overview of S.eps in Evaluation Process 14 

Ctiapter VI 

Figure VI- A t Relation Between Recommended Subtest and 

Outcomes for Grades 8-12 • 68 



ERIC 



CHAPTER I 
INTRODUCTION 



Since 1970 the concept of career education has gained considerable prominence 
in the eyes of educators and the public at large. Career education legislation 
has been passed by state legislatures and the U. Coi^ress. Funds have been 
appropriated at the federal, state, and local levels tc implement career education 
programs. The U.S. Office of Education has appointed an Associate Commissioner 
for Career Education. 

The concept of career education achieved this prominence because of a pervasive 
sense among educators and laymen that U.S. education is in need of reform. 

Career education, which is generically defined a% "the totality of (educational) ex- 

V ' 

periences through which one learns about work, is seen as an important response 
to that call for reform. 

The career education concept has achieved prominence considerably faster than 
It has achieved precise definition. It is prominent, but it is stlU in the process 
of being operationally defined. In some ways it is analogous to the concept of 
"mental health. " Mental health is widely held to be desirable; it has been 
incorporated into local, state, and federal legislation; public officials have been 
charged with the responsibility of fostering it; and programs have been funded to 
help people achieve it. As with c&reer education, however, operationally 
defining and measuring "mental health" is still the subject of debate and is likely 
to be for some time to come. 
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To piirsur the analo«y one stop further, the concept of ''mental health'* hf^s 
achieved its current nieasure of specificity because individuals, at various 
points, decitied to Impose a priori but logically defensible definitions. They 
then sought to evaluate thv extent to which treatments designed to achieve these 
dimensions of mental health were achieved, and to evaluate the ultimate Impact 
on the individual of having achieved them. Similar issues are germane to the 
concept of caroor education. It Is the judgment of many Individuals at all levels 
v\ho have worked with the concept of career educatimi over the past several 
years that the time has come to deal with these Issues and to systematically 
refine the concepts 

The purpose of this monograph is to report the results of a series cf actions 
that were taken to achieve this objective during the summer of 1974, and to 
lujjKest a strategy based on these actions A^hlch can carry the process of definition 
another step forward. It should be made clear at the outset that the decisions 
reported in the pajjes which follow are viewed as tentative, but essential, steps. 
They are tentative- in the sense that thos;? who made them do not claim they should 
not be altrr^'d over time. They are essential in the sense that until some 
decisions were made and tested, the process of operationalizing the concept of 
career education could not proceed. 
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An ObjectivcS'Based Di^finition of Career Education 

During the first half of 1^174, the Office of Career Education of the U.S. Office 
of Education ensa^ed in an Intensive effort to specify the objectives associated 
with the concept of career education* The effort tised as Its base theoretical 
constructs emanatinu from the work of industrial psychologists and educational 
theorists largely bes^lnnln^ in the l^^Os, Also considered were several recent 
attempts to dc-iini! operationally career education, and the experience of programs 
which have been in operation over the past severa't years designed to Implement 
career education in public school settings. More specifically, the following 
sources provided the basis for the results emanating from those efforts: 

• theoretical works pertaining to career development, career maturity, 
occupational maturity, and individual learning and development; 

• federal legislation and policy pertaining to career education; 

• state leuislation and policy statements regarding career education; 

• descriptions of state and federally funded efforts to develop career 
education curricula and efforts to assess the results of utilizing such 
curricula; 

• Hf'^^criptions of federal, state, and local efforts to Implement career 

orlucatinn prourams in public school sytitems In all types of settings In 
each of the 50 states; 

• bonks, articl/^s, and othor publications which dealt with the concept and 
definition of c^i't or education; and 

• a sf'rii s of « (injs nnd ronfori>ncei3 with state and loiral individuals 
O „ engaged In implen^enting the concept of career education. 
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tiasfd on tht sf so\iri 4 i*, a set of objectives commonly held to be associated with 
the oart t r itltuation concopt emerged. In all, 33 objectives were identified and 
grouped into nine n-^ajor categories or outcome areas. Once developed, the Set of 
objectives was reviewed by individuals of varying backgrounds who were engaged in 
implenifnting rare«T ««du«*ation efforts. The objectives were generally found to be 
acceptable by thtise individuals. 

It should be noted that the set of student objectives does not Include statements 
regarding acquisition of technical skills associated with specific careers or occupa- 
tional citiaicrs. While providing students with technical skills (e. g. , carpentry 
skills, clerical skills, etc. ) should be an aspect of a comprehensive effort In 
career education, implementation of the career education concept may not current- 
ly effect the content of vocational courses or the level of proficiency of the students 
enrolled. Rather, it is presumed that implementation of career education will 
currently provide a contextual framework for the courses which may affect the 
numoer of students enrolled, may effect a change in the type of student enrolled, 
and may lead to an increase in the number and type of vocational offerings. The 
experience of the district in placing exiting students has been included as one mea- 
sure of the appropriateness and the qviallty of the specific pref»ration opiK>rtunitle8 
being provided young people in the overall career education program. 

Thf results of this process are presented as Table 1. In essence, these objectives 
are rr>nHider< d to be central to the career education concept and to constitute an 
obj<rctives-l>ased operational definition. It is recognized that further refinement 
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o£ the objectives aUtements may be necessary. They represent current best 
thinking in this regard and should be viewed as the starting point towards a 
realistic operational definition. 

It should be noted at this point that much of the literature, and even such federal 
legislation as Part D of the Vocational Education Act, to the extent that they define 
career education, combine both objectives associated with the concept and pro- 
cesses presumed to be associated with achievlf^ these objectives. In fact, some 
definitions of the concept focus entirely on the process dimensions. Specifying a 
set of educational processes associated with the conceit as part of the operational 
definition was considered and rejected. While this eventually should occur, it was 
concluded that it would be inappropriate to do so at this time. 

Wfiat follows on Table 1, then, is but one facet of what should eventually become a 
full definition of the concept of career education. It is presumed that based on a 
knowledge of how these objectives are best achieved, the process or treatment 
dimensions will be added. 
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TABLE I 



TAHKEH KD^CATION OBJECTIVES IN TERMS OF 
STUDENT OUTCOME STATEMENTS 

I. Students will demonstrate increased self awareness: 



A. Students will be able to describe their own current abilities 
and limitations 

I5. Students will be able to describe their own current interests 

and values 

C. Stiidonts will display positive attitudes toward themselves 

D. Studvnts will recognize that social, economic, educational, 
and cultural forces influence their development 

II. Students will demonstrate increased competency in basic academic/ 

vocational skills:-'* 



A, Students will be able to demonstrate generally useful numeri- 
cal skills 

U. Students will be able to demonstrate generally useful commun- 
ication skills 

C. Students will be able to demonstrate generally useful infor- 
mation processing skills 

D. Students will be able to demonstrate generally useful decision- 
making skills 

£« Studi^nts will be able to demonstrate generally useful inter- 
personal skills 

IIL SiMdents will demonstrate increased awareness of work values and 

pu^ijcsb ct deijire to engage in paid and/or unpaid work: 



A. .Students will recognize the ba^^cs of various work values 
''i. Students will possess positive attitudes towards paid and un- 
paid work 

IV. StM'lents will demonstrate increased awareness of and knowledge 

about v«.f;rl'; 

A. SHidonti? will know the major duties and required abilities of 

/Uffc rt iit typrs I paid and unpaid work 

*^:;^»; r:- '.V;.. * t* ^ r-*, ♦h^t providing Rftid^tf with cdfi^^^r sktlN {^. , rarp^niry, typing, arcounting, etc. J 

asi't* t»*fH w I'l. ^p^i. ifii vot/itttmfi t« an tfifei^ral aspect of a cimprehensive effort in career eduratii»)< it wai deci^d to 
ett< t irl» 4fafrrr ««nfs tf^^rdift^ achievement ef «uch sXiiltt frorn thte list of f^Jdent outcome because currently a large 
Q rr tif-r c^rf^pf ndMratian efforts da not address this area directly. It should be noted that Raceme At <^ students has 
PDI/^pi; inrxudfd 
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Students will know dificrerii in work conditions and life styles 
a:550ciate(l with different types of paid and *'rxpaid work 

C. Studcntd will know entry requirements for major types of paid 

and unpaid v ork 

D. Student:? will know the impac:t of social and technological change 
on paid and unpaid work 

E. Students will Know the important factors that affect work success 
and satisfaction 

V# Students will demonstrate increased competency in career decision- 

inakini; sl ills: 

A* Students will be able to associate their own abilities and limitations 
with possible success in present or future paid and unpaid work 

B. Students will be able to relate their personal interests and values 
to types of paid and unpaid work and their associated life -styles 

C. Students will be able to identify, locate^ and utilize sources of 
information to solve career decision-making problems 

D. Students will be able to determine the potential for future 
advancement/personal growth in work of ir choosing 

K. Students will know the steps to be taken and the 
factors to be considered in career plannir^ 

h\ Students will dctnonstratc active involvement in career decision- 
making 

VI. Students will donicmstrate good work habits: 



A. Students will be able to plan work effectively 

B. Students will be able to adapt to varied work conditions 

C. Stiidents will have a positive attitude towards the concept of 
quality in relation to a work task 

Students will have a positive attitude towards conservation of 
environtnuntiil and hun^an resources in accomplishing work tasks 
K. Students will have a positive attitude towards responsibility for 
tht ir own bcijavior and accomplishment of self-imposed work 

tasks 

I '. St idcjjts will di inonstrate a desire for continuous learning, both 
in .H«'hool and out 
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VII. Studf-nts V. ill dcinons*^ rate work- st?el<ing and work-getting skills: 

A. Students will be able to identify, locate, and utilize 
sources that contain information about existing paid and 
unpaid work possibilities 

B. Students will be able to demonstrate skills required in 
applying for and accepting work 

VIIL Students \/ho are leaving the formal education system will be 
successful in being placed in a paid occupation* in further 
education, or in unpaid work that is consistent with their current 
career education. 

IX. Students will be aware of means available for continued education 
once they have left the formal educatioival system: 

A. Students will be able to identify sources of additional 
education in major types of paid and unpaid work 

B, Students will be able to identify means to support additional 
education for themselves in major types of paid and un^ 
paid work 
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Because of the pragmatic ne^d of school districts to res{K>nd to questions posed by 
those interested in their efforts to implement career education and the equally 
practical need to further its definition^ there currently exists a demand to assess 
the results of efforts to implement the concept. The main body of this monograph 
provides an operational approach to meeting these two needs. It outlines steps 
which a local school district can follow in conducting an evaluation of its career 
education effort. A longer term result of following these steps would be the 
accumulation of information necessary to determine the extent to which the objec- 
tives associated with the concept have been achieved; that is, the baseline data for 
determining whether or not ihese are individual attributes which should be fostered. 
In addition, the result would provide the data necessary to determine the most 
effective ways of achieving these objectives. 

It is recognized that few if any school systems have explicitly attempted to achieve 
all of the objectives associated with career education. It is also recognised that 
resources at the local level are limited. Thus, what is provided is a framework 
for assessing efforts in career education rather than a tight prescription. If the 
framework is followed and if the results are shared with USOE, it is expected that 
both local needs for assessment and national needs for policy ihformation and 
further theoretical development will be met. As a significant by-product, it is 
further expected thai the administration of career education efforts will also be 
improved. 
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The chapter which fuliuws contains a schematic and narrative overview of a 
suj4j4cHtcd approach to evaluating career education at the local level* The 
remaining clyiptors will be elaborations of this overview^ In preparing this mono* 
grapli, it was roconnized that levels of experience with educational evaluation 
vary greatly. As a result, some readers may find it somewhat difficult whUe 
others may iitid it disappoiniingly simple. An attempt was made to achieve a 
middle ground with the understanding that some readers may choose to skim 
some i hapern while others may choose to followup on the footnotes and biblio- 
j^raphy which provide ureater depth. 
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CHAPTER II 
PI-AN FOR EVALUATING CAREER EDUCATION 

Evaluation is not an end unto itself* Some authors refe.* to evaluation as a part 
of the planniny process. For purposes of this monograph it will be discussed as 
part of the management process. This process is viewed as beginning with the 
development of a plan to reach certain ^oals or obtain certain results. Next the 
plan is iniphMnented and then the results are evaluated* Ideally, planning the 
evaluation sh<iuld Oi cur after tlie goals have been set and plans for achieving them 
laid out, but before those plans have been implemented. 

Based upon the results of the evaluation, the original plan is revised or Improved 
as appropriate; the revision is implemented, and its results are evaluated* From 
this perspective, evaluation is simply the process of examining present perfor- 
mance in order to improve future performance. 
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. t ^ pr»i\ ith ashintancr in nieotins two 
: I v.o basic assumptions 

t - 1 1 l:tt* i! uuifU iinos. Thv primary 
» ;t ■ \r<MiM 1/r f a s s i s ta Hcc to program 
. in !.>rn)\ilatin^ evaluation designs. 
Ml .itili/. fl at thi- local level and should 
:rin; r abii\niieii th'^L each local 

. - m! p >t< ntia^ ' a'^uo at the national level. 
< : p. trHnslatinu specific local objectives 
! 1 rn n^ilat ion is necessary in order for 
UMw'il 1» \ < ) to utili/.<' the diverse local 

• i(> hi n i<»ntinuouH or cyclic process. 

» ]'< < xprMi f! to provide the basis for program 

H i f » t't . 

uHior. plan prc^sentrd here is primarily 

;t s ( vahiate their efforts in career 
. : w! liir plan in to provide information 
» ;H t?-f local level, }t is also designed 

V v.!tU es.senti;ii iiifoi'J'i/ition. These two 
I ' • M * . e led to th<- sopn rat ion of 

20 



13 



At the broadest level, there are four phases in evaluating any program: 

apfcDfyini; what is U) hv evaluated; designing the evaluation plan; implement- 
ing the plan; and reporting the findings. 




Within each of these ff'ur broad phases of program evaluation there are 
several steps which should be completed in sequentUl order. More 
8pcc:ifi<ally, the steps which make up the problem definition and design 
phases dcfim- (and therefore must precede) the remainder of the effort. 
Thus, these initial phases must make clear what is to be evaluated, 
why it is to be evaluated, how it is to be evaluated, and how the findings 
are to be reported. Because they are so crucial to the overall process, 
they are the focus of the major portion of these guidelines. 

To provide an overview of the entire process of evaluating career 
edtication in the context of a local school system. Figure 11- A specifies 
\3 aU-Uf. or t.',si.s whic h should be eon^pleted. The first two of these 



steps may be considered as the pre-design or problem definition phase 
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(Phase 1, page 13 ). Steps 3 through 10 are those necessary to complete 
the evaluation rfesign (Phase 2, page 13 ) as they specify a procedure 
to solve the evaluation problem. Steps 11 and 12 relate to implementation 
of the design (Phase 3, page 13 ). Step 13 (Phase 4, page 13 ) 
involves reporting the findings on which program modifications may be 
based, leading to renewal oi the evaluation process. 



OVERVIEW STEPS IN EVALUATION PROCESS 
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A narrative overview of the 13 steps is provided below: 
Phase One « Problem Definition 

The first step is to prepare an overview of the program. This defines in 
general terms what facet of the school system is and is not to be evalua- 
ted. It should be completed by school staff as the basis for negotiation 
with external evaluators or as the basis for determining what external 
assistance^ if any, is needed. Preparation of the overview is the 
subject of Chapter IIL 

The second step is to complete a series of tasks leading to completing an 
Outcome Quostion/Trratment Group Matrix. In completing thrse ta^s the expected 
relationship between the studextt outcome objectives of career education, 
the activities or treatments which will lead to these outcomes, and the 
students who receive these treatments in the project are identified. 
Essentially, this is a further specification of the narrative overview. 
In some situations* it may be advantageous to involve external evaluators 
at this point in the process. While the use of external evaluators ia 
not essential^ if such assistance will be used, this is a desirable point 
at which to include it. Preparation of the matrix is the subject of 
Chapter IV. 

Phase Two - Evaltiation Design 

The third step is to formulate the evaluation questions. There are three 
possible types of questions: those pertaining to student outcome objectives, 
other program objectives, and program management. Kor example, the 



''^Some programs, e. g» , Part D, require their use. 



»t idem utile type questions might seek descrif^ions of the extent 
to which 1 u? outcome was attained, descriptions of the extent and quality 
{A treatn^entst and an assessment of the relationship between treatment 
and outcome* The basis for formulating student outcome questions was 
laid in the preceding step; formulating the questions is discussed in 
Chapter V. 

The fourth step is to specify the evaluation design* While tliis should be 
implicit in well-formulated questions, it is helpful both to those attempting 
to relate results of several evaluations, and to clarify thinking at the 
local level, to make basic decisions explicit regarding the need for pre 
and post measures and comparison groups* Design considerations are 
disc issed in the context of question formulation in Chapter V# 

The fifth step is to specify the data sources to be used to answer the 
evaluation questions* Some possible sources of data are: those afflicted 
by the career education effort(s), those who cause the effects, and 
observers of the effects. For student outcome questions selection of 
<^}.ita sources and data collection instruments are discussed in Chapter VU 
[ or treatment questions data sources and instruments are discussed in 
Chapter VII# 

The sixth step is to select or develop data collection instruments* 
T or several outcome questions and most process or treatment questions, 
:rjt;tr jn:entation mtist be developed specifically for the evaluation effort* 
inswer some student outcome questions, a limited number of 
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instruments are available and are presented for consideration in Chapter 
VI. 



The seventh step is to prepare a sampling plan. The details of the 
plan depend on decisions made in preceding steps and administrative 
considerations. The guidelines in Chapter VUZ suggest poixits to consider 
in making sampling decisions. 

The eighth step is to prepare a data analysis plan. While detailed analysis 
decisions ought not be made until after data is collected, provision for 
data processing and an outline of the airticipated plan should be prepared 
at this point. Data analysis is the subject of Chapter DC. 

The ninth step is to prepare a data collection plan. Here the tasks, 
personnel, and target dates for implementing the design are specified. 
Preparation of a data collection plan is the subject of Chapter X. 

The tenth step is to finalise the design and prepare a reporting plan. 
All decisions made to this point are reviewed in terms of technical 
feasibility, cost, and the needs of users of the evaluation findings. 
Once this is done and modifications made» as necessary, a plan for 
reporting the findings to the various users, along with target dates, is 
prepared. This process is the subject of Chapter XL 
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Fhase Three - jinplemcntation 

The eleventh step is to implement tbe data collection plan* It is presumed 
that problems will arise and all will not go as smoothly c.s envisioned. 
It is important to record all deviatioct^ from the plan as vn aid in 
analysis and interpretation*. 

The twelfth step is to analyze the data collected. The first task is 
processing and tabulating the data. Then suitable analytical techniques 
are selected and analysis is performed. Steps eleven and twelve, as 
well as reporting, are the subject of Chapter XIL 

Phase Four Report Findings 

The final step is to report the findings and make recommendations as 
appropriate. The report should be prepared in sections on the assumption 
that not everyone will have need for all sections of the report. As an 
aide in comparing the fiixlings of evaluations across projects, tbe report 
should provide basic data indicating the context in which the cjireer 
education effort was carried out. 

ft should bo noted that the emphasis throughout is on obtaining student 
outcome and related treatment data. While other possible types of 
evaluation questions are not discussed e>q>licitly, e.g., management 
processes, the process outlined is equally applicable. Once the question 
has been formulated^ the steps to be accomplished are the same. 
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B. Design Work Sheeta: An Aid to Plaimmg 

Aa an aid in iniplcmcnting several steps in the evaluation plan, and as a 
summary o£ the key decisions which may be of interest to readers of 
the final report. Table II-A contains an evaluation design work sheet* 
This work sheet is designed to provide a summary of the evaluation 
plan in a convenient and logical format. The last step of the evaluation 
design process described in this manual is to complete this worksheet* 

The worksheet is provided here at the beginning of the discussion to 
provide the reader with a preview of the system* It also provides a 
handy index and set of instructions for later use, as instructions for 
completing the work sheet are included in the appropriate columns as 
well as chapter references* A separate sheet<s) should be completed 
for each student outcome area* 

The reader is asked to briefly review the logic and format of the 
worksheet at this stage. £ach of the following chapters relate specifically 
to it* 



ERLC 



^1 -M 



y 



y I— 



< 



> 6 T 



— c 



r ^ ^ 



"-^ 

^7 i 



? 5 u 

*15 



— c 



JUT, 



B *J 



«8 S 



S 5 0 iJ 
S 2 « w 



•p4 



if 



96 



z 

I 

< 



w 



< 



I 

< 



8» 
U 

I 



-0 




20 



I 
1 



■c 



■3 



ERIC 



2b 



21 



CHAPTER m 

STEP I; PREPARING AN OVERVIEW OF THE PROGRAM 

To design an evaluation, there muat be something to evaluate. Simplistic 
a- this may sound, the first problem facing an evaluator is to determine 
what there is to study. Whfle with this step it is not necessary to describe 
the program in great substantive detaU, it is essential to establish tb- some- 
thing exists which is distinguishable as a discrete subject for the evaluation. 
The purpose of the overview, therefore, is to convey in simple language to the 
designer of the evaluation, and ultimately to the reader of the final evaluation 
report, just what it is that is to be evaluated; that is, to briefly describe 
the subject of the evaluation and to provide some supporting, background, and 
contextual material. The overview should be considered as an orientetion 
document for which the evaluation team n\akes clear what is, and is not, 
to be included m the evaluation* 

To be roost helpful, both in developing the evaluation design and as a draft of a 
portion ol toe final report, the overview should address in broad terms the 
questions of; 1) what are the purposes of the career education effort, 2) what 
are the major activities undertaken to achieve these purposes, and 3) what 
resources are used. The questions should be answered for the period to be 
covered by the evaluation. 

*If outside evaluator s are to be used, the overview should be included in the 
materials on which evalu&tors wiU base their techni^^al and cost proposals. 
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If the program is in the planning phase when the evaluation effort is designed, the 
overview should be based on expected inputs, processes, and program objectives. 
If the program is on- going, the overview should set forth the expectations for the 
coming year. Care must be taken to insure that the overview is not based solely 
on the expectations of prior years. Typically, i< is not appropriate to utilize 
sections of program proposals, annual reports or other previously prepared docu- 
ments as the overview, although such documents will be a major aid. If there 
are significant breaks with the past, these should be noted. 

It may be helpful to conceive of the overview as an executive summary 
of the program to be read by the school superintendent, the general 
public, and interested officials at the national level. In more concrete terms, 
the overview should contain the following information in summary form: 

• Origins of the program. This section will discuss the background 
of the entire program; how it got started; its original goals; and 
how it changed over the years. 

• Program goals and objectives. This section will state and 
discuss the present goals and objectives of the program and how 
they are interpreted by the staff. Program priorities will also be 
discussed. 

• Staff and student participants. This section will discuss the 
number and types of staff members involved in the program. 
The responsibilities and duties of each wiU be summarized. 
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In addition, the number of student participants will be given by 
appropriate sub-^ groups such as grade level, course enrollment, 
school, etc. 

• Major program activities. This section will summarize the major 
program activities and discuss how the staff and student partici- 
pants are involved. This will provide an understanding of how the 
program goals and objectives are being met. 

• Amount and source(s) of funds. This will be a listing of incoming 
funds and major types of expenditures. This will provide an 
understanding of the size of the program and its emphases. 

The overview need not, in fact ought not, be lengthy or require much time in 
preparation^ It should be a terse statement which defines the subject of the 
evaluation. From the overview the reader should be able to understand whether 
the entire system is to be included in the evaluation or, if not, what portion is 
included. The reader should be able to understand whether all efforts which 
might be considered as contributing to outcomes of students associated with 
career education are to be considered in the evaluation or whether only selected 
efforts are to be included and the basis for making that selection. It should be 
clear to the reader how long the efforts have been going on in the school district 
and who is responsible for them. Finally, it should be clear who is providing 
services and who is receiving them. 
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The importance of clarifying these basic evaluation issues at the start of the 
process cannot be overstressed. The term career education is subject to 
multiple definitions. Many of the objectives of career education may overlap 
with objectives of other programs or emphases of a school district* Activities 
designed to lead to achievement of career education objectives may be supported 
financially from several ^sources and staffed by personnel from various areas 
in thfr district, 

A recurring problem in the evaluation of career education efforts since 1970 has 
been to specify what aspect of the school program is to be covered* In some 
situations it may be appropriate to define the subject of the evaluation in terms of 
a particular funding source or combination of funding sources* For example, if 
a school district is supporting career education through funds received from a 
federal Rrant, state career education funds, and local revenues. It may be ap- 
propriate to include all activities regardless of funding source. M^at is most 
important in terms of the evaluation design is that this decision be made explicit 
at the start of the design process* 

Of equAl importance to the design ia making explicit those broad program 
activities which will be included in the evaluation. For example, most school 
systems have taught vocational skills courses for decades. In many districts 
work fxperience {cooperative education, distributive education, etc. ) courses 
have been on- going for years. Clearly, vocational skills courses and work 
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e3q)erience programs address many of the objectives of career education programs 
and thus legitimately might be included in the scope of the evaluation. In some 
school districts these programs have been added or expanded in recent years as 
a result of a commitment to career educaUon. In some districts the quantity of 
effort has remained constant for many years but substantive changes have oc- 
curred recently and may be associated with career education. In other districts 
there have not been changes either in the quantity or the quality of vocational 
courses or work experience programs but since they are conceptually related to 
a comprehensive career education program they are defined by the district as 
part of their career education efforts. In designing the evaluation it is essential 
that it be specified at the outset whether such activities are to be included in 
total* included in part, or excluded from consideration. 

Closely related to the need to specify which, if any, long- standing efforts of the 
school district will be included in the scope of the evaluation is the need to 
specify what new activities are to be included. If the primary focus of the 
e-/aluation is in terms of funding sources or use of specific school personnel, 
then a decision must be made regarding the inclusion of related activities. For 
example, if a career education coordinator devoted some time to developing a 
proposal for state funds, should the activities under this grant be included? 
Similarly, if the evaluation is to focus on a particular program supported by a- 
specific grant, should additional activities also be included in the evaluation that 
were by-products of the particular program but are not supported by the grant 
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or are not staffed by individuals directly associated with the Rrant? 

There are no right or wrong answers to the above questions which can be 
generalized to all situations. What is important is that they be addressed directly 
at the start of the evaluation process. If they are not, considerable time ir.ay be 
wasted in completion of the succeeding steps and it will be extremely difficult 
for readers of the evaluation findings to be certain that they understand the basic 
nature of the effort which was evaluated. 

Once the overview of the program is complete, it should be possible to move 
easily to the next step in the evaluation process and complete a series of tables 
which make the broad statements of objectives, activities and participants in- 
creasingly more specific. 
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CHAPTER IV 

COMPLETION OF OUTCOME QUESTION /TREATMENT GROUP MATR IX 

The primary purpose of this framework for evaluation is to assist LEAs in 
evaluating their efforts in career education. Use of this framework should 
contribute to the hody of career education escperience by permitting 
educators to accimiulate experience and compare the effect of different 
techniques and approaches. The accomplishment of these broad goala requires 
the translation of local program activities and objectives into sets of common 
terms which makes such accumulation and comparison possible. While it is 
recongized that local differences may make such translation difficult, it 
is important to do so to provide a basis for strengthening the develop- 
ment of career education* 

In this chapter a process is described which results in the development of a ma- 1 
trix which describes the basic characteristics of the career education effort being 
evaluated. In completing the tasks leading to preparation of the matrix, progriun 
information will have been organized in a manner which greatly facilitates com- 
pleting subs'squent steps in the evaluation process. Thus, while the matrix 
itself will be of primary value to those interested in comparing efforts in 
several locations, and to local people interested in obtaining an overview of 
the program, completing the tasks leading to its construction are crucial 
in developing the evaluation design* These tasks may be summarized as folloHS^ 
• Specify career education treatments associated with those activity 
or treatment areas applicable to the local situation. 
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• Identify the student receiving each treatment. 

• Identify groups of participants receiving the same treatment(8). These 
will be considered the "treatment groups." 

• Select the student outcome questions appropriate for each treatment 
group and complete the Treatment Group -Outcome Area Table. 

• Select the more specific student outcome questions appropriate for 
each treatment group and complete the Outcome Oiestion/Treatmest 
Group Matrix. 

Each of these five steps is discussed In somewhat more detail in the pages 
which follow. 



ERIC 



3ti 



29 



A« Specify Treatmenta 

The product of this task should b« a list of the career education treat- 
mexits or activities provided students* The broad limits of what will 
be included on this list were established in preparing the program overview 
(Evaluation Step 1, see Chapter m). For eacan^le^ if it was determined 
that the evaluation should encon^ass all career education activities 
within the entire school district, then all treatments provided to all 
students should be included* On the other hand, if the decision was to 
limit the evaluation to certain types of activities or to those funded by 
a particular source, then the range of treatmeits listed here should 
reflect that basic dec is ton* 



In identifying the career education treatments reference should be made 
to the 12 activity areas listed on the following page In the left hand 
column of Table IV-A (Activity- Treatment Table)* These 12 brw«d areas 
were taken from legislative and policy statements regarding career educa- 
tion* They are judged to be on the one hand sufficiently broad to encom- 
pass all treatments being provided to students, yet sufficiently distinct 
to be meaningful as broad categories. The list of career education treat- 
ments should be arranged next to these broad headings. Table IV-B 
provides an illustration of a broad area, elementary familiarization, as 
it relates to specific, local treatments. 
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table: iv-a 
activity - treatment table 



ACTIVITY AREA 


SPECIFIC TREATMsNTS 


Elempntary School Familiarisation 




Junior High School Familiarization 




Senior High School Familiarization 




Elementary School Career Orientation 




Junior High Schoc . Career Orientation 




Senior High School Orientation 




Elementary School Career Exploration 




Junior High School Career Exploration 




Senior High School Career Exploration 




Junior High School Work Experience 




Senior High School Work Experience 




Placement ( exiting students) 




Other Activities (indicate) 









SI 



TABLE IV-B 


Illustration 
Activity* Treatment Table 

Activity Area «5p*^r.ifi*T Trf'fltmi'nt 


Elementary Familiarisation 


Im Curriculum infusion 
2. Career education field trips 
3« Visitors in the classroom to talk 
about their jobs 

4. Use of special career materials 
provided by '*X" 

5. Group sessions with the counselor 


Junior High School Familiarization 


1. 




2. 




3. 




4. 




5. 



& is important to consider throughout this task that a career education 
treatment is by definition something that happens to studextts which is 
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distinguishable from other treatments provided by the scIkksI system* 
For example, if field trips are cited as a treatment, they should be 
a particular type of field trip (defined in terms of puriK>se, type of 
advance preparation, or some other factor which distinguishes career 
education field trips from others). 

Several approaches to identifying career education treatments are possible* 
Given the complexity of znost career education efforts, however^ eicperlance 
has shown that a broad abased approach involving planneis, administrators, 
and evaluators is more effective than more limited approaches involving 
only administrators or only evaluators* 

While it may not be possible to describe ^ of a project's activities in 
terms of student treatments, every effort should be made to do so* 
In some cases the relations h^ between a major project activity aiul a 
student treatment may be somewhat indirect but « nonetheless, possibla and 
reasonable to define. For example, the local project may provide in- 
service training designed to improve an elementary teacher's ability to 
utilize career education techniques in her regular classroom activities* 
At first glance one might consider t^ aspect of the local program to 
be an '^other*' activity* Upon careful consideratiox^ however, one would 
conclude that the in* service training was designed to change the teacher's 
rlassroom behavior in order to have a new effect upon students; for 
exanplc, to familiarize them with the world of work and the range of 
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opportunities for work. Further, this training for teachers can be 
e3q>ected to result in student outcomes different from the outcomes esqwcted 
if the training did not occur* Therefore, it is possible to describe this 
activity as a treatment even though at first glance it did not appear to be. 

Although such questions will not be discussed explicitly in this monogriqph« 
a whole set of important e validation questions may be formulated based 
on the need to assess the effectiveness of such program aictivftfes cr |ioaBtt«B. 
The process of formulating andaiswem^ such questions wcmld be the 
same as are discussed in this and following chapters* 

B. Identify Students by Treatment 

Once the list of career education treatments is complete, the next task 
is to identify the popiUation receiving (or expected to receive) each 
treatment provided by the program. For each treatment, the partic^ant 
population should be identified with sufficient specificity to enable an 
evaluator to xfentify easily the specific partic^ax^s for testiag or 
other study purposes. 

The ideal r.. Hhod would be to list the names of all ^rticipating students 
by treatment. This is often not practical, however. More feasible may 
be to group studejsts by grade, course, school, or some other grot;^ 
designation. Experience has shown, for example, that many programs are 
able to identify all sixth grade students at schools x,y, and z as 
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receiving a particular treatment* In another program it may be 
necessary to be more specific. For example, the students in Ms* Jones' 
third grade class, school X and Mr. Smith's fourth grade class in the 
same school are those which have a curriculum infused with career 
education concepts. 

The format for reporting the participaxds by treatment is very straight- 
forward and shown on Table IV- C telov. It is simply an extension of the 
Activity- Treatment Table discussed in Task A. 



TABLE IV- C 

AC TI VIT Y- TREATM ENTS-PAR TICIPANTS TAB L E 


Activity Area 


Treatments ^ 


Paitidnams 


Elementary' 

Fa?nlHari/atii>n 


1. Curriculum infusion 


Glides 4, and 6 at &xnvn, Jones^ and £mltis 
Elementarv Schools 


2. Career education tietd 
trips 


urade 4 at )oncs Elementary 


i, Visitors in the classroom to 
talk about their iofas 


Grades at Brov^n, Jones, and SmitVi 
Elementarv Schools 


4. Use of special career mate- 
rials twovided bv 'X'* 


Grade 3 at Smith Elemei^ary 


3. ciroup se^lons with the 
counselor 


Grac^ 5 and 6 at Brown« loses* and Smith 
Elementarv Schools 


Junior Hi^h School 
F imilhiri/ittion 


1« Curriculum Inftssion 


Grade T social stUilies in Main Junior l-Ugh 



orn I able A. ^ ^ 
a should be recognized that in some situations this may be one of the 
most difficult tasks in the entire evaluation process. Without it, however, 
it will be impossible to implement the evaluation. 
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C, Identify Treatment Groupa 

Baaed on the table relating treatmenta and participants, the next task 
is to identify groups oi treatments being received by the same students. 
The result of this effort will be a list of discrete student groups defined 
in terms of the manner in which they participate in the career education 
effort being evaluated. The sum of the number of students in each group 
should equal the total number of student participants in the program. 
In other words, care must be taken to insure that the same student does 
not appear in two groups. 

In some situations, experience indicates that very few groups will emerge 
while in others the list may be quite extensive. To acme extent the 
number of groups will depend on the specificity of the treatments listed 
in the first task. Jt is recommended that the level of specificity reflect 
the type of treatment but not judgments regarding extent or quality. 
The issues of extent and quality of exposure to various treatments are 
best handled as questions to be addressed by the evaluation (see Chapter V: 
Evaluation Questions). 

Depending on the situation, completing this task may be a relatively 
simple or a highly complex undertaking. Essentially, its completion 
involves inspecting the Activity- Treatment- Participant Table (IV-C) and 
forming discrete categories of students based on the treatments they 
receive. Table IV- D which follows illustrates the results of this process 
based on the information contained in Table IV-C on the previous page. 
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F'AnTJnrANT - f.nt gkoup tablf. 


Av ttv ilV 

A ffii 


Trfat:j4«-nr» 1 Pa-.' .parts 
■^.M- T«>^K' !V.^ ) 1 f L^n*..' !V.r) Treatni*«nt Groups 






Gr<i<U'<t 4, S» and b at Bro\%n, n| A. Grades S li 6 at Brown, 
Ji^m'f, and Stnilh Etemen* Bp Jones, li Stnlth 
tary Sctiools.* G Eletnentary Schools.* 


2, Car*»i»r *-<l\ication 


Gra ie 4 a: Jones Elementary B. Grade 4 at Jones 

QK Elementary 


ro.'Hi l*"* iA\t< about 


('ifAiir^t t-^^ <tt Broun. Jones. Sf C. Grade 4 at Brown and 
an-l Sm.ih Klenu'Atary « Smith Elementary.* 


4, Lsp tif sp<-& ;cii tArrer 
rr.atert;4N provined 
fay 'X- 


Grade 1 a: 5mith Fleinen- « D. Grade 3 at Smith 
tary. IB8 Elementary. 


^, <jru«tp kesiiions with 
the counselor. 


Gradt-s ^ jnd 6 at Brown, 9K E. Grade ^ at Brown and 
Jone9, ai.d *Smith Eicmen* 5! Jones Elementary, 
tarv SciionU. ^» 




F, Grades 1 2, at Bron^n, 
|S Joneft. & Smith 
f« Elementary, 


NOTF ln«p^ctiv/n of th** participant f oluii.n rr\ al« that Ah»l^ aU 4thp 5th aD*! 6th graderft receive 

t arriculuni Infusion, " only the Jonrs 4th grader* go on field trip*. As a result* the treatment 
xroup roltui^n Tvfieci^ thr nerd to differentiate among the 4th grade particlpanta. 



The example above was chosen because it represents a slightly more 
complex configuration than is typically found at the elementary level. 
In many situations, inspection of the treatment /participant table will 
yield only two or three groups of students. For example, if the table 
had indicated tha*^ all elementary students went on career education 
field trips, the number of treatment groups would be reduced fronn six 
to four (group B and C would be included in group A, "grades 4,5, and 
6 at Brown, Jones and Smith"). 

h\ some situations, particularly at the upper grade levels, it may not 
hf". possible at this point in the evaluation process to be certain precisely 
huw students arc (or will be) grouped according to specific treatments. 
In high school, lor example, this would require knowing the schedvile 
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of each individual student. In such a case, the most feasible procedure 
is to estimate, on the basis of informed judgment, how treatments are 
likely to cluster and then to verify the estimates during data collection 
and analysis. Aa a basis for formiilating evaluation questions and 
completing the steps which follow, developing a careful estimate of the 
number and nature of treatment groups is an extremely important task. . 
In addition, from a program planning and management point of view, 
the discrepancies between these start-of-the-year estimates and the 
findings of the evaluation may lead to modification of the effort (or 
expectations) during the next school year. 

D. Complete the Treatment Group -Outcome Area Table 

Based on the results of Task C, the product of this task will be a table 
which relates participants by treatment group, to the broad student 
outcome area(s) appropriate for that group of students. In Chapter 1 
a set of 9 student outcome areas sub-divided into 33 more sfMcific 
outcome objectives associated with career education were presented. 
These currently are considered as central to the cimcept of career 
education, but are not presented as necessarily exhaustive or in final 
form. They do, however, represent the current best effort in this 
regard and as such are the basis for the pro^Msed evaluation approach. 
. The nine broad student outcomes areas presented as statements in 
Chapter 1 (Table 1} are summarised on the following page, 
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I. Self Awareness 

11. Basic Academic/ V'ocational Skills 

III. Aw.ircnoss of Work Values and Desire to Engage in Paid and/or 
Unpaid Work 

IV, Awareness of and Knowledge about Work 

V. Competency in Career Decision- N'aking Skills 
VI. Good Work Habits 

VIL Work-Seeking and Work-Getting Skills 
VIII. Placement 

IX, Awareness of Means for Continued Education 
The Treatment Group-Outcome Area Table which is the product of this 
task consists of a matrix followed by an indication of the outcome area 
appropriate for each treatment group. An illustration of a completed 
Treatment Group- Outcome Area Table, based on the information 
contained in Table IV-D, is presented below: 



TABLE IV- E 
1 REATMENT GROUP-OUTCOME ARhA TAiJLE 

ILLUSTRATION 
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Elemcnt*ir- Familiarization 
1. CurriciiltiiT! liiitslon 


X 


X 


X 








Ffelil Tri^s 




X 












X 


X 


X 


X 


X 


X 


>^>ccin' Materials 








X 






5. Croup Cnu:>^^f ioff 


X 












'mfm/i///////j////m//////M/////////f^^ yjyy* / w ^j'm 


Area 


I 

II 


I 
n 


I 

II 


1 

TI 


I 

u 


I 
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From Table IV<-E on the preceding page it is possible to see at a glance 
(a) the various groups of students effected by the career education effort 
being evaluated, (b) the specific treatments provided each group of 
students, and (c) the student outcome area{s) expected to be effected 
by the treatments provided each group of students. Where groups of 
students receive treatments related to several activity categories t the 
table should be constructed so as to reflect this situation. Table IV-F 
provides the basic shell for completing this task for all projects. 
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Tk5.ATr!ri^; i group - outcomf area table 

BA.SIC SHELL 



Activity Ari n; 




Trontmcnts 



Activity / «•'■-'' ■ 



1 
2 



Treatments 



Outcornr- Ar<»a 
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. Conipletc the Outc otne C uestion /Treatment Group Matrix 
On the Outcome Question/Treatment Group Matrix the specific student outcome 
questions to btf an:»vvered by the evaluation associated with each broad student 
outcome area are identified and must be related to the appropriate treatment 
Rro;ips. The completed matrix provides a summary of the specific outcome 
questions appropriate for discrete groups of students participating in the 
career education effort being evaluated. This matrix and the Treatment 
Group -Outcome Area Table on \vhich it is based provide the detailed descrip- 
tive information necessary to complete the subsequent steps in the evaluation 
design* More specifically^ taken together they provide the information needed 
to formulate the final evaluation questions, to develop the sampling plan» 
and to make initial decisions regarding data analysis* 

The matrix consists of rows denoting student outcome questions based on the 
nine student otitcome areas and the 33 related specific objectives presented 
in Chapter I. In all» 36 student outcome questions are included as rows of 
the matrix. The columns of the matrix represent the treatment groups - 
which constitute the local career education effort* 

To complete the Outcome Question/Treatment Croup Matrix, the column 
headings must be filled in locally* These will be the same as the headings 
of the columns of the Treatment Group-Outcome Area Table* Once this is 
complete, a checkmark denoting the related outcome area should be placed 
in the appropriate cell of the matrix. Finally^ the specific question(s) 
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for each outcome area should be selected and a mark placed in the appro- 
priate cell. 

Utilising the information provided in Table IV-£ (Treatment Group-Outcome 
Area Table) and decisions regarding objectives, an example of a completed 
Outcome Question /Treatment Group Matrix is presented below as Table IV -G. 
In this example, it is presumed that the special materials used by the Smith 
elementary 3rd graders are specifically related to decision-making and that 
a change in students ' ability to make decisions is an explicit objective only 
in this facet of the program. The example further indicates that the objective 
of producing more positive attitudes tov/Brd themselves is applicable to all 
treatment groups and that increasing numerical and cMnmunication skills is 
appropriate for all groups in grades 3 through 6. 



TABLE fV.G 






n 




tommrd Bmlnm 


X 


X 


X 


X 


X 


X 


•kUla 


X 


X 


X 


X 


X 




n * B Xiic r6a««4 commuafM- 
iiou skills 


X 


X 


X 


X 


X 




n« 0 lac riMed d^cU imm 
maUsig skills 








X 







Table IV-H which follows presents the shell of the complete Outcome 
Question /Treatment Group Matrix. 
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jbflit>' to «k*<vribe their dwa 



Do fttitJetfis display mof^ 

plHltiVtf atfitUtitfS tOHifd 



D, H4Vtf -.nUentN increase their 
ftfco^nitiM thai social, 

wulfural {orces inflnvnce their 
tievtflopment? 



A4:4ii<ftiisc VcHjattOn^l 



A. HAve fhidentf Increased 

thpir level of {generally useful 



H« fl.ivf« ^fmli»nf'; im^iv-i^i'd their 
i:ort}niuttii-4ti«>n iktiK? 



l4»vel v»f ^i-ni'rjllv iHefiil 



fijve >tuijtfnts mwreiseii their 



ffavi? itiiiliMits iticrcs«;ed their 
J nt^'fMtTv.-n il .kill.* 
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TABLE IV. H.. cont. 



BEST con MMUBtE 




theif rr*rk.H(nttion of the* Ki»e« 
of v4ricu» work \«tluo«? 



Paid ATul/ot Unpaui 
Work 

tm^sltiVf" 4itiii|(i«'v tCMjrii 

inud and tinp4id wch4v7 



of and Knowledge 
About Work 

A. iiaive studisins im ro t trtl 

Lhtf m;i|or duties and required 
abilities of oiftcrtftU ty|H£» of 
IMiA 4tui unpaid work? 



B. Hav« 9tu«<c*^f Increased 

tlielr knowledge of differences 
In work condltior^t and tife 
ityUt associated with 
different types of paid and 
unpaid work? 



H4Vft< students )ncr4*>i»ed thoir 
knimled^e of i?nfrv rcquinfments 
for mjior types of p^id and 
unpaid Hork? 



^.novvledftt? of thi* trrti^ct of 
soo.mI ami tvchsK^io^fral 
vlijn^«" in p^d and unpaid 

Wt3fk ' 



Ea ll.ivt.' ■itidi.-nt'i increased thoir 
knowlcd^jtf of th* injpcrtdnt 
tJCfors that aft'trirt v\Ork 
*iiri*t?vs jihi s.itisfdiTtiOQ? 



ERIC 



51 



TABLE IV.H .. com. 




r>cci«i iTt. Making 

ihrit Ability lo ^»«»'»4'Ciatp 

Itfnttattona *\t%\, ^^i^nsbtr 

in prt ><W(t or 
futufr pat'i Arl gti;»Aid 



Ikurir ability tu r*'l«Al« 
thrir p^rfiOTUit inli-rfflta 
and valuei to typ^s of 
paid and urpaid ^\<irk and 
th<"lr associated liff -styles^ 



C Hdv« acudents in<.r«>aa«d 
th^tr ability to (j^ i^^ntify, 
(b) locate, and Um utiU^e 
sourcef of inforn^tton to 

ntaking probl«'m«'* 



D, Have students increated 
th«ir ability to d<-lermine 
th^ potential for future 
advancemenl /p^rMonal 
grov^th in work of their 
choosing'* 



Hdvp ttudentf increaaed 
the^r krt»>wtedt^** nf th^ 
atep^ to be takm and the 
factors to be considered in 
car**i r planning ^ 



JUv#» students* Im r#»ased 
their active involvement 
in career dects ir,ft-mak4ng'' 



TABLE IV-H -^'cotU 




VI. Improved Wor> 

A. Are itudtnti Ablt to pl^a 

work effectively? 



B. Are itudenti more adAptAbIt 
to varied «rark iltufttloao? 



C. Do itudente have a more 

pnftitive etiifitde to«»rdl the 
concrptf of quAlity ia roU- 
tion to ft work tAtk? 



Do studentf have a more 
po»itive attitude toward* 
conservation of envlroa- 
mensal and human reiotircea 
ta accompUthing work taeka? 



E. Do etudente hatre a more 
positive attitude towarda 

r£;*f«Mii*Ibr.Uy for thetr owo 
behftvier and accompHehmeat 
of seif-im|K»sed work taekf ? 



r. Du etudettts demofietrate an 

increaeed dee ire for continnoua 
learning both in ichool and out? 



Vn. Increased Work Seeking and 
Work Getting Skilla 

A. Hftve ttodentf i per eaead their 
ability to la) identJ^p ih) 
lAraf#, a«Hi ic\ ••flll^e aourcea 
that contain information about 
pftid and unpaid work^ 



Hav» itudente increaeed their 
2«vf:; of akiUe reqislred in Ca) 
applying for« and (b) apceptiey 

work? 
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TABLE IV-H ^ont. 



IT OR nmitMiE 



46 




vd or afff* pngagfd in 
^hts Campari* a 4th prior y^ftrf* 



B. Uuw m«&y st»4*'^ftt9 have lM«n 
*^ rtnhd occupation* 

with prior vf«r«? 



C. Of thof# pU'-^^ in (a! further 
education, and ih^ employment, 
hpvk many con«id«r the place* 
m<-nt to be coft»i»tenS *ith 
their career plan*'' 



Of %ho99 not placed in fufther 
e<1ucatt0n or in a paid occupa* 
tton, how many are engaged la 
U» unpaid «-ork ronaialent with 
their career plans, and how 
«ice^ thla rotr^are veiih pricr 
yeara** 



IX. Inert* aaed Awarenf 119 of Vcans 
for Cofttifiued education 



{(ave students Increaaed their 
aUUty to i<»^mify fotsrcea of 
additional <»da/*atlon in major 
typrtf of patH and unpaid work' 



lUvi? studenta increaaed their 
j>>fht'' to Idi^ntify meana to 
*<upport a'fdtttonal education 
for th**rri*<»|vr ji in ma^r types 
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CHAPTER V 

FORMULATING THE EVALUATION QUESTIONS AND 
SPECIFYING THE EVALUATION DESIGN 

A comprehensive evaluation of career education an a 8ct»ol district may be 

expected to address questions dealing with three areas of concerns 

• student outcome objectives; 

• other program objectives; and 

• program management* 

For each area, questions may be posed which seek descriptions of some 
aspect of the program or which seek explanations for descriptive statements* 
While subsequent discussion will focus on formulatiz^ and answering questions 
related to student outcomes, the ssme basic considerations apply to the other 
areas of interest* 

In completing Step Z of the evaluation plan, the questions pertaining to the 
achievement of student outcomes were identified* In addition, the base was 
laid for formulating questions regarding the extent and quality of treatment 
exposure for the students from whom the outcome measures will be obtained* 
As a result, the base has also been laid for formulatii^ descriptive and 
explanatory evaluation questions. The formulation of both descriptive and 
explanatory evaluations questions cannot be separate from a consideration of 
rcst^arih doHiun. Tiie remainder of this chapter 
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KST WY AVMLABLE 

tluTcfo- •: ^ v.-iri.'r^ ih- .tfiTi of dcsign, foUowed by further discussion of 

The baaic design of the evaluation study ts dctated by the specific form of the 
evaliiatiufi questions. The use of relative terms in the question, such as 
students exposed to the program will increase their knowledge of..." or 
"students exposed to th& program will become more positive toward.. •" 
necessitates the use of a pre /post design, since relational or difference terms 
of this type require at least two measures, one of which is taken prior to 
program exposure, or prior to exposure to some significant portion of the 
pro^ratn; the other measure would, of course, be taken after such exposure. 

Since the career education programs take a significant amount of time to 
administer, or, to put it in other words, since program exposure must 
occur c/er a significant period of time even under the best of circumstances, 
simply taking pre /post measures of the exposed students and calculating the 
differences between these measures would not necessarily indicate the effects 
of the program. This is due to the fact that duriug the interval between the 
mcasMres, the students undergo other "treatments" including continuing 
maturation and the synthesizing of experiences. Thus, a change in the 
measures cannot loiiically be attributed to the program. 

The tsiiu.ll solution to this design problem is to use a control group; this is 
to -itaiu the same pre/post n^easures, with the same time iiAerval between 
their., iron, students who have not been exposed to the program. The simple 
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logic of this approach of course demands that the jion-e^sed student be 
identical to the exposed students in every respect other than program «q>osure, 
or that they not be different from them in any systematic way which might 
affect the results. For example, if the program-e^wsed students were 
brighter or less bright than the non-exposed students, or if they were from 
different socio-economic classes, one could not attribute test score differences 
between them to the difference in program exposure. The best way to 
minimize systematic differences between the two groups is to assign students 
to program exposure and to non-exposure on a random basis. Unfortunately, 
as is the case with most education programs, this approach is not feasible 
for career education programs. Not only can students not be assigned ran- 
domly, but in many programs it is difficult to assume that non-exposure is 
possible - either because the program encompasses aU of the students or 
because teachers and students r olved in the program are likely to have an 
iirfluence on teachers and students who are not actually participating in the 



program* 



Since it is unlikely that a satisfactory control group can be defined in advance, 
other potentially applicable design approaches should be explored. The 
most feasible approach is one which selects students, or classroom units, 
randc-mly. and obtains information on each student's or each class" exposure 
to the program. In this way, one can identify the level or "amouiA" of 
program exposure for each student or class, and treat those with the lowest 
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exposure level as a coniparison group relative to those with the highest exposure 
level. Comparisons between the gains made by the high- exposure group and those 
made by the low exposure group would then provide the basic data needed for 
assessing whether or not the program had had any impact on the outcome measures, 
providing that there are meaningful differences between the treatment received by 
the high- exposure group and those received by the low- exposure group. If the 
treatment variable is quantifiable on a fairly fine scale, sue! as number of hours, 
the "high" group might consist of the 25% of the students who had the greatest 
number of hours and the "low" group might consist of the 25% of the students who 
had the fewest hours of exposure. 

If the treatment differences are in fact meaningful, the design would lend itself to 
analysis tq determine the strength of the relationship between treatment and out- 
comes. The framework fcr this analysis is described below. 

The importance of the design in determining the relation between treatment and 
outcome can be realized by examining the following figure: 



Treatments 


Outcomes 


T 4 C 

T = C 


T ^ C 


T = C 


1 


2 


3 


4 



An adequate evaluation of a program is possible for situations falling into cells 1 an 
2. In these instances, the treatments will be different for at least two groups, 
designated as Treatment (T) and Comparison (C), and the outcome s will either be 
different, as in cell 1, or not different, as in cell 2. {Cell 2, which applies to 
programs in which the treatments were different but the outcomes were equal would 
cover situations in which treatment was either inappropriate or not sufficiently 
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potent to realize the desired outcomed*) Cell 3 would apply tc those programs 
ior which the treatments are equal but the cHitcomes are different. In. this 
case^ £ioniething apparently was$ operating to change student or teacher 
behavior, but specifically what treatments were operating is unknown* Cell 4 
includes those programs that show no difference in treatment and no difference 
in outcomes between Treatment and Comparison grotq>s« This case may be 
the result of a poorly conceived and/or poorly implemented treatment. 

Three questions that should be answered in any treatment*outcome eviduation 
are as follows: 

Im How do the experiences (treatments) for treatment and comparison 
students diffar? 

2. How do the outcomes (products) for treatment and comparison 
students dilfer? 

3« How do the treatments relate to student outcomes (in instances 

in which there are outcome differences between treatment and compari- 
son students)? 

An answer to the first question provides for a decision on^which row of 
Figure 1 is appxvpriate. If the classroom e^eriences do not differ for 
children in treatment and comparison classrooms, then the evaluation need 
go no further^ because any observed outcomes would have no program^* related 
e:q>lanation. It is important to emphasize that the differences are xuA 
hypothesized differences but actually observed differences in process* 
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Many a proj^raiv. had phinncd to implement a certain processy later to discover 
that ilic proposed strate^ied were not iniplemex^ed, resulting in little, if any, 
difCcrencc* between treatment and comparison classrooms. 

Given that there are observed differences between treatment and comparison 
clab*s»rooni:i, the next question is whether outcome differences exist* XC 
outcome differences do not exist, then the process/product aspect of the 
evalttation need go no further because, although the treatments differed between 
treatment and comparison, the appropriateness or |K>tency of the treatment was 
not adequate to influence students* bei^vior (e.g., self concept, career 
awareness, etc* Assuming that tl^re are both treatment and outcome dif- 
ferences between treatment and comparison groups, then the reXationsbip 
between process and product can be empirically tested (e.g., by using 
correlational analysei>). B: is not adequate to sixr^ly demonstrate product 
differenced and exposure differences between comparison and treatment 
groups and then assume that the differences are related. Upon empirically 
testing the relationship between treatment and product differences, potential 
causal infl leni es can be identified and later manipulated to test causal hypotheses. 

Application of the above outlined approach would provide answers to many 
of management's questions, including (1) what is the nature of the program, 
(i) whAf- gains in student outcomes were realized, (3) why did the treatment 
^roup ^ain more than the (jomparison group, and (4) what process dimensions 
Tiho'iUl be! cntphai$i/-ed in subsequent years and what dimensions should be 
terminated 
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From the foregoing it should be clear that the basic evaluation design will be a 
function of the topical substance of the evaluation question, i. e. . does the question 
require pre and poit tests, and the possibilities permitted by the context in which 
the evaluation takes place, i. e. , is a comparison group available. The final 
formulation of the evaluation questions should reflect the decisions regarding both 
topical interest and feasibility. For example, while most of the "evaluation" 
questions posed by USOE imply the need for pre- and post- test measures, they 
do not address the Issue of comparison groups. As a result, flnal formulation of 
the outcome q lestions requires locd decisions in this regard. 

In practical terms, the situation with regard to comparison groups is likely to 
resemble one of the following three possibilities: 

1. No comparisons possible (al) students receive identical treatments; no 
relational analysis possible). 

2. Untreated comparisons {pure control groups, i.e. , there is a group of 
students who are comparable to those receiving career education treat- 
ments in all major respects except receiving the treatments). 

3. Contaminated comparison (no control group but there is a range of exposure 
among students in treatments). 

Each of these situations is related to a different form of evaluation question. For 
CMmple, a general question associated with student outcome area I (increa'sed 
self-awareness) would be formulated as follows under each of the above conditions: 
1. Have students exposed to the program increased their ability to describe 
their own current abilities and limitations? 
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2« Have students exposed to tbe program increased their ability to 
describe their own current abilities and limitations to a greater 
extent than those students who have nc^ been exposed to ti^ program? 
3» Have students exposed to different program treatments increased 
to an equal extent their ability to describe their own current 
abilities and limitations? 
Primarily as an aid to those attempting to summarize evaluations of career 
education from several locatr ns^ but also as an aid in clarifying tbe process 
at the local level, it is advantageous to specify which of the design alternatives 
was selected* In terms of the design work sheets presented in Ch^Aer 
this simply means recording in column 3 which of the three alternatives with 
respect to comparison groups is selected. In addition^ if for some reason it is 
impossible to take pre- and post*^ measures for particular outcmne areas, this also 
should be indicated, and the evaluation question would then be re<- formulated to 
reflect this design chanjse. 
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CHAPTER VI 

MEASURING OUTCOME VARIABLES 
SPECIFYING DATA SOURCES AND INSTRUMENTS 

Once the evaluation questions have been selected and the design specified, the next 

step uill be to specify for each question the sources from which data are to be 

collected. The determination of the data sources for each question will be a functiai 

of where the data exists, and the availability of existing instruments to gather that 

data. 

This chapter discusses the use of various data sources. The chapter also introduces 
several data collection instruments that are currently available and attempt to 
measure career education program outcomes. A table is presented showing the 
relationship between these instruments ^nd some important student outcomes. This 
is followed by a brief discussion pertaining to the development of tests locally. 

A. Data Sources 



It is helpful to classify sources of data relative to student outcomes into two 
categories? (a^ direct and <b) indirect. Direct sources are those which manate 
from the students themselves. Indirect sources are those which have their 
origin in some outside source possessing knowledge concerning student outcome 
|e,p., teachers, parents, employers, etc.). 

In the context of an evaluation effort, direct source data may be divided into two 
categories. First, that data which must be obtained directly from the students 
Hurinis the course of the evaluation. Second, that data which have already been, 
or will be, collected from students and stored elsewhere for reasons apart from 
the evaluation. 
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1. Direct- source I)ala 

In performing an evaluation ^vhich seeks information concerning student 
outcomes it will be necessary to collect some data from the students them- 
selves. Ffowever. in terms of efficiency, it la desirable to maximize date 
collection from secondary sources. Thus below we first discuss secondary 
data sources, 
a. Secondary Level Data 

Even though all outcome data could be sought £r(xn the student, it is 
frequently easier and more efficient to obtain available data from a 
secondary source. Consider, for example, the situation in which 
program personnel desire to know how many students have been placed 
in a work situation during a given period. While that information could 
eventually be obtained from each student, it would be unnecessary to 
invest the time and effort in surveying each student if suitable records 
on such placements were available. It is important, therefore, to first 
determine what information has been, or will be, generated elsewhere 
that will be helpful in evaluating the Career education effort. 

More specifically, the suggested approach to assessing the extent to 
which students have increased in several dimensions of academic/ 
vocational skills (outcome questions U-A, and II-Bt is to utilize the 
. '•^'Stilts of the achievement testing program regularly used by the school 
system. While such tests might not be given on a pre-post test basis, 
< oMiparisons can ho rtiado^a) between participating and non- participating 
Q Students based on the amount of change between a prior year's test 
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results and the results of testing during the evaluation period, th e 
on a modified pre- post basis, (b) sirnply between a group of 
participants and a matched group of non-participants» a post*only 
design; or ic) between matched groups of students differentiated by 
their extent of participation* also a post- only design* 

Other information that is normally produced by school systems will 
also provide indicators ufieful in the evaluation of a career education 
program. For example t aptitude test scores, results of interest 
inventories, and attendance records may all provide meaningful 
information in assessing the impact of Career education upon students* 
The key to using this kind of information lies in having a clear identifi- 
cation of the program participants* While this may seem obvious, 
experience has shown that one problem which e^^luators frequently 
experience is that of identifying the treatment population. 

In summary, it should not be necessary for the evaluation to duplicate 
already existing data collection efforts* Existing data should be used 
before additional data are collected* 

Primary Level Data . 

It will not always be possible to obtain direct data from secondary 
sources; in fact, it will be necessary to obtain most of the information 
directly from the students* This may be obtained either through the 
use of instrumentation already developed or through the use of ''home 
made'' instruments* As a rule, the utilization of well- constructed and 
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appropriate instruments is preferable to the development of new and 
untried instruments. If for no other reason, there results an obvious 
time saving. A discussion of existing instrumentation and guidelines 
for the local development of tests of student outcomes is presented 
later in this chapter. 

2. Indirect Data Sources 

Not all indicators of student outcomes need be obtained from the students 
themselves. If, for example, a program decides to use improved school 
attendance as an indication of improved work habits, the data source might 
be school or teacher records and net the individual students. 

In seeking information concerning student placement, as another example, 
there are several possible sources of data. One source might be 
employers. The program might seek to survey area employers to 
determine how many students have secured employment. A second 
approach would be to obtain information from students themselves. A 
mail or telephone survey of students to determine their current employ- 
ment could be conducted. 

Experience indicates that there are problems in obtaining data from these 
sources. It is difficult to obtain sufficient returns on mail surveys from 
employers to provide useful information. Followup visits provide better 
information but require consic arable investment of time and money. In 
all but unusual cases, the costs are prohibitively high. 

erIc *^ 
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Attempts to ^Ican placement information from school records (e.g.. 
counselor or placement office) more often than not fail to Improve upon 
the above techniques. School personnel frequently must go from one task 
or problem to another, leaving the recording of their efforts for later* 
When time for recording action is available^ recall is difficult. Th« 
information generated in this manner is usually grossly estimated 
(not remembered) and of limited value* 

Of the three" approaches, however, the latter is the one most likely to be 
used* It is both the least expensive and most managable approach for the 
school system* since the data source will not be the students themselvesi 
but rather, school records* In designing forms to record information 
from such records it is necessary to determine what information will 
satisfactorily provide the answer to the given question, and to consider 

the nature of the records themselves* In some situations it may be 

« 

desirable to develop new reporting forms to replace those currently in 
use. It should be noted^ for example* that regular reporting of events 
and actions generally provides more reliable information than does post- 
performance data collection. It is one thing, , for example, in seeking the 
number of students placed in full-time employment, to ask personnel 
'*Ilow many students did you place in jobs last year?" and quite another 
for them to submit weekly data sheets containing this information for the 
past week. 
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In securing infornation concerning student outcome measures which are 
best obtaine<l from school records or from regular reporting of school 
personnel, the follcwing procedure should be followed: 

• Identify the quesiions to be answered and the sources of the information 
{e. g. , Question: How many s'.udents were placed in part-time paid 
employment during the school year? Source: school counselor and 
work experience coordinators', 

• Develop a reporting format where one does not already exist. 

• Develop reporting procedures as necessary (e.g., weekly report to 
Career Education coordinator). 

• Implement procedures at beginning of period to be evaluated. 

Following ♦hese piocedures will not only insure more reliable information 
for purposes of evaluation, but also has the potential of producing some 
useful management information for program directors. The latter benefits 
would be enhanced if the reporting format would also provide for noting 
various demographic facts about each student* such as age, sex, and 
ethnicity. This would also be of value in analyzing other outcome flndings 
and in studying the relation between treatments and outcomes, since the 
program n.ay be having different effects on different demographic groups. 
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B# InstrumenUtioo 

!• Available Instrumenta 

Referring back to direct data aourcea, there are several data collection 
inetmmenta which attempt to measure student career education outcomes. 
During the summer of 1974, USO£ undertook a review of these instruments* 
While the review was not exhaustive, it was esctensive and is presently 
continuing. A review panel was convened for the purpose of reviewing 
instruments that might be appropriate measures of career education 
student outcomes. 

In preparation for the paners work relevant instruments were sought from 
as many sources as possible* The search for instruments included a 
review of the evaluations of all first round Part D programs and other 
career education programs, A survey of test publishers was also 
conducted and selected third party evaluators were asked to suggest 
instruments that had proved to be of value in their past efforts. In 
addition to these sources, local and federal personnel engagisd in the 
operation of career programs were asked to make recommendations 
of instruments to be reviewed by the panel. 

The search produced 90 instrtiments for consideration by the panel. The 
tables which follow display the results of the panels efforts and should 
be of value in designing an evaluation of a career education program. 
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The first table (Table VI- A) is concerned with instruments reviewed by 
the panel and recommended as appropriate for measuring the student 
outcomes of career education efforts. Table \I»B lists the career 
cJucalion areas for \i^hich no tests were recommended. 

The third table in this section (Table VI- C) provides specific information 
for each of the recommended instruments. The last table lists tests 
which uere found to have some potential in measuring student Outcomes 
but were not recommended because of insufHcient psychometric data. 

Table VI- A which follows, entitled ''Instruments Recommended by Panel, 
shows the relation between the various parts of the recommended 
instruments and various student outcomes. The table is sub-divided into 
three grade level groups, 4^6, and 8-12; applicability for Grade 7 

is noted separately where appropriate. 

As the remarks section of Table VI* A (and t!ie summary matrix shown in 

Figure VI-A indicates, most of the subtests or parts of the recommended 

instruments were found by the panel to be measuring more than one 

outcome^ This means that the scores obtained in such a subtest cannot 

be interpreted unambiguously. If, for example* there is a gain in the 

scores for a subtest ^^hich measures outcomes A and B, it is impossible 

to tell whether this represents a gain in outcome A, a gain in oticome B, 
or a gain in both. 
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In instances \vhere the multiple outcomes nneasured by a subtest can be 
viewed as meaningful parts of some higher level composite concept, the 
subtest may be viev/ed :ib measuring the composite* By way of analogy^ 
an instrument judged to measure length, width, and height simultaneously 
would be judged to be measuring volume. Since volume is a composite 
of these three factors, and a change in any one, any two, or all three 
would necessarily result in a change in volume. 

In theory, a gain in the scores for a subtest which measures two (or more) 
outcomes may also result from a gain in one outdome and a loss in the 
other (or others). The obvious implication of these points is that 
preference should be given to those subtests wMch measure only one 
outcome, followed by those which measure two outcomes that are clearly 
related to each other. 
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TABLE VI-B 



Outcome Areas For Which No Teats Were Recommended 

(Grades 8 - 12) 



I. 


A. 




B. 


n. 


A. 




B. 




C. 




D. 




E. 


111. 


A. 


V. 


D. 


VI. 


A. 




B. 




C. 




D. 




E. 




F. 


VII. 


B. 


VIII. A. 



III. A. Recognition of bases of work values 
V. D. Determining potential in preferred work areas 
Effecti\« work planning 



VII. B. Skills for applying for and accepting work 



B. Placement in work 

C. Placement /plans consistency 

D« Unpaid work/plans consistency (unplaced students) 

IX. A. Identifying sources of career- related education 

B. Identifying means to support career- related education 



^ Since it was assumed assessment of these outcomes would utilize the 
results of the school districts' existing testing program, instruments 
pertaining to the outcomes were not included among those reviewed. 
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THE ASSESSMENT OF CAREER DEVELOPNffiNT <ACD) 

PublisheJ by iknsghton Mifflin Sales Office: MeAsar«m«td Cui4ABc« Oc^. 

Grade levels! 8-11 BMiM^^MMtkamietu 02107 

AdmiRlstration time: 125 mimttes (three fessions) 

Cost: Test boddets (package ^ 3S) - $ 13.95 
Answer sheets (package of 3^ - $ 3. 75 

Scoring' Cannot be scored locaUtf excep by apeclal arraafetneot; aaimiianr data lor Ave ai^calea and 

42 additional items are reported. 

Remarks: 

a. Answer sheet {tfovides for collecting responses for up to 19 locally coostrocted qMStloes* 

b. Should not be used below 9di grade excef^ for giottps hsvii^ above average volml atdUt|r* 

c. Subgcorc key 

Suhscore 1 • Occupa^oiml characteristics 
Subscore 2 - Occupational (^paraticm requirements 
Subscore 3 • ExplcMtory occupational exp^enceSp general 

Separate scor» for eadi o€ the foUowfi^ dtst^ 

• Social /health f ainl Personal Setvl^ 

• Business Sal^ and Management 

• Busines Operations 

• Techwlogies and Trades 

• Natural, Social, and Medical Sciences 

• Creative ami Af^ed Arts 
Siib^ore 4 - Career Planning Kmwlec^ 
Suhscore 5 - Career Planning Involvement 



CAREER MVELOPMENT INVENTORY (CDI) 

By Donald £. Super, et al. Teachers College, ColumUa University, New York, N. Y. , 1(X>27 
Available from authors^ to be used with their permission. 
Grade levels; 8*12 

Administration time : 30 minutes for average studei^; all students stoidd comply Imnt^ in one dass period. 

ScCiing: Can be scored locally or commercially. 
Remariis: 

a. There io 9e% stereotyping in a fetir items, but emplrieaf sti^ea shew Umn la no aex blaas revision to 
remove the stereotyping Is la process. 

b. Scale key: Scale A iHanning Orientation 

Scale B ^ Resoirces io^ EscplMation 

Scale C - Information andT>ecislon>Making 

CAREER EDUCATION QUESTICS^NAIPE (CEQ) 

PuUished by: Minnesota Research Coordinating Unit Vocational Edt^tioa, 
Univenity' of Minnesota, Minneapolis, Minnesota 

Grade levelsi K*3, 4-6, 7*9 

Administration time: 30 - 45 minutes 

Scoring: Can be scored locally. 

Remarks: 

a. Each ^radc level test yields one ovcfall score. 

b. OuaUty of picture's in the K*) form Is not uniformly high; vocabulary level may be high* 
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TABLE Vl-C cont. 



CAREER MATtIRm' ISA-ESTC^V (CMI) 

Published b> : C TB McCr^%v Hill, Del S\OfAe Research Pari*, Monterey, CalifiMHia 93<MO 

Cr lit* levels. 7 - 12; items should be presented mlly to those midestfs who are not aUe to rea«1 at the sixth ftrade level 

AJmhis^tfati^M) time: 2 \ 2 Hotirs 

Cost: Test tx*i4.JeU {PJcKa^e of 35> $20.00 
An^er <iheet< (Package of S5) $ 4,00 

Scoring; Can he scored lOi-ally or commercially 

Remarks 

a. The Attitude Scale items may permit the <tev^opmeiit ai meaninnsful subicmt. 
MiAi^r tir^f^f oi tex strr^otypifm; no clear «vi4c&ce of mtm bias. 

A - Altitude Scale 

Pjri I » Knowing Vounelf (Self*- Appraisal) 

Part 2 ' Knowii^ Aboid |ohi (Ocei^^ti<nial luformatloa) 

Pan '^ Chpositu; a lob {C^ Selection) 

Pan 4 - Toi^irrj; Ahead (PtannliHt) 

P;irt > - What Shoidd They Do? (Problem Solving) 

(p3K S ^as not recomtrended by* the review panel due io ifisagreement regarding the scoring 
i>f ^ome M the items) 



mmRENHAI APTITUDE TESTS uith Career Planning Program (DAT) 

Puhtishi»d by The ps> chological Corporation, 304 East 45d> Street, New Yoik, N, Y. 10017 

Grade fevets. H - 12 

AdmioiktratiOh time: 235 mjnuxcs 

Ci^t: Test twKleli {ptcka^e of 2S| SO 

AnsvN.T sheet^ J^ackai^e of 50) $12. 50 {IBM and Op Scan) 
Aiisu.T Nluu»ts ^vi*:ka;.t? of SO) $1 i.(X) (NCR) 

S*:onii^ Can K.' ^woreJ locjlly or commercially. 

Remarkjb; 

^- r^r^^r V:^nn,ng Program t« carr^Rtly tormaUled on an tadii^ridual Mttdent •^•f^^Hi^n'' pr^P^'ed 
ur/iUT V^rSip »4mi?-ry of findingti th* m0^\ for thu aaiwnary la •ho*n in App«nd4« D. 

h. M{^hl he utilircd in lieu of achlevemeitt testing for oiMome 1! A and 1! 6. 
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TABLE VI- C cont. 



SELF OBSERVATION SCALES (SOS) 

PuHIUhtfd NjtlotuI Ti?slitiA StfA'icc, isi^ liTrwin Pi'ad. Durham, Ncrlh Curiiliiw 27705 
Grade levels: K-^, 4-6 
AJministratidD time: 20-25 mlnuii'^ 

CiMt; Test IwkJeU aiul Ar^uer shifts <f>dcka^e of ^0) » $35.00 

Scoring; CdTUK^ he scisred tiMrally* 

Remarks: 

A , This li»5t Has recommeiKkjd by the review panel only on the eondiUon that the {mbli&ber fSWiU^ with 
information re$;ardin4 the )>corin^ kcyu and weights for the subscaleStiSA*^ Aii/endkn C f4*r pultUshrr's commrfii*. 

b. Local review fcv set^dve Items was' recommended fay the review ^oet. 

c. The review panel recommended further worK b^' the puUlsher on validation. 

d. SuJbCide Key ; 

K-^ fPrimaf\' Level) 

Subscale 1 - Self Accej^aoce 
Substrultr 2 - S<:«lal Mati^ty 
Subscale I - School Affiliation 
Sutvcalc 4 - Self Sccurit>' 
Subicalc 5 - Achievement Motivation 

(Subscale 5 was not recommended by the review panel due to 

Insufficient f^chometiic data* ) 

4-0 ^Intermediate J«e>'<*fl 

Subscale I - Self Acceptance 
Suhscale 2 - Self Secmlty 
Suhscale 3 - Social Maturit>' 
Suhscale 4 * Social C<mfldence 
Subscale 5 - School AfHliation 
Suhscale 6 - Teacher Affiliation 
Sulscale " - Peer Affiliation 
Subscale 8 - Achievement Motivation 

{Suhscale H was not recommended by the review pane! due to 
insufficient |*<ychometT<c data. ) 
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TABLE VI-D 

n s I S ^^A VINCI puomisinc, approaches but insufficient 

iiKVt LOPMENT OR LACK OF PSYCHOMETRIC DATA 



New Mexico Career Education Test Series: 

Monitor, P. O. Box 2337 
Holl>^ood, California 90028 

Orientation of Career Concepts: 

Evaluative Research Associates^ Inc. 

b444 Floussant 

St. Louis, Missouri 63121 

Affective Assessment Questionnaire: 

Minnesota Research Coordinating Unit for Vocational 

Education 
University of Minnesota 
Minneapolis, Minnesota 

Decision Making Scale: 

ffiEX 

ZSib Erwin Head 

Durham, North Carolina 27705 

PSE Career Education Test: 

Policy Studies in Education 
52 Vanderbilt Avenue 
New York, New York 10017 

Attitude Toward Self and Others Scale; 

Center for Occupational Education 

North Carolina State University at Raleigh 

Raleigii, North Carolina 

Carter Awareness Inventory: 

Cornell Institute for Research and Development in 

Oct upaiioiial Education 
State University of New York 
Ithaca, New York 80 



TABX^ VI-D (Continued) 



-Scile: 

Scho^ District of the City of Royal Oak 
4000 Crooks Road 
Royal Oak, Michigan 



Locally Developed Instruments 

For outcomes not measured by these instruments and In planning to 
secure information from other indirect sources such as teachers, 
counselors* parents, employers, etc. , it will be necessary to consider 
the development of instruments locally. If It is decided that the need for 
one or more "home made" tests justifies the time and costs Involved 
in developing such tests, there are several important technical factors 
which must be considered. Some of these factors apply to each Item 
to be constructed; others apply to the "test" as a whole. In either case, 
the major considerations pertain to the need to maximize the validity and 
the reliability of the students' responses, and ultimately of the score or 
scores which will be generated by the test. 

The starting point for maximizlr.g validity is a clear delineation of the 
subject matter, or outcome area, which the test is supposed to measure, 
followed by p Usting of specific content areas which the participating 
teachers plan to cover, or have covered, within the outcome area. The 
test items would then be constructed around the specific facts or concepts 
which represent the more important teaching objectives. 

Having specified the contents of the test items, a decision is needed 
concerning the format to be used for the items themselves. There is 
ustially a choice among several alternatives, such as True/False, 
multiple choice, Agree/Disagree, and matching, among others. It is 
more difficult to construct items in some formats than in others, but the 



effort may be worth while. For example, multiple choice items arc 
more difficult to construct then true/false items, but the results arc apt 
to be more stable and more valid since there is a 50/50 chance of 
getting a true/false item right simply by chance. (Where there is a high 
probability of being correct on the basis of choice alone, test makers 
generally construct a relatively large number of items as a way of 
compensating for chance effects and insuring an adequate spread in the 
scores achieved by the respondents.) In addition to the ease or 
difficulty of constructing the items, the choice of format should also 
be based upon such factors as; the suitability of the format for the 
intended respondents, the length of time to administer the test using each 
format, and ease of scoring. 

In the course of constructing the items foi the selected format, great 
care must be taken in the wording of each item, and in the case of 
multiple choice items, in the wording of the response alternatives, to 
insure that there is no ambiguity and to insure that the vocabulary and 
sentence structure are within the capabilities of the potential respondents 
The language of the item, as well as the test formats, must be so 
designed as to insure that incorrect responses occur only because the 
student does not know the correct answer, rather than as a result of 
difficulty in understanding the item or confusion with regard to the 
test directions. Both of these factors may be checked by means of a 
TDal!- scale prc-to.st of thr instrument involving 20 in 30 students. 
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; ri* - if jro iiu'ludes as wide a range of ages and abilities as 

, :rsi vii rr isin^ items Lu insure that the langw?,|ie is clear and 
t;c , It - t ur^v i',> lose yi^ht of the specific knowledge which the 

ir . is: itMl tft .^fjsrss; often one finds that the item is easy to 
; :> it dot s i ot have direct relevance to the outcome area* 
? ' ^ i.n rron occurrence may be minimized by haviog cognizant 

: - V ii^Mn ii>r appropriateness of both the substance^ 

I ^.1 the itcMiis. and the specific language and format being used. 

)o iM ai.so rt'\'iew the test as a whole for such defects as 
r. .f cutJteiii areas, inadvertently providing the answer to an 
i.ujthrr item, and under- sampUng or over- sampling of the 

V lacth and concepts involved ^n the teachers' treatments* 

iv;try. the developn:ent ol adequate tests is a complex^ time 
Hit' iktidertakina. Wnile there clearly is a need for instruments 
idcquatoly address the student outcomes associated with career 
f-hmiu^Mii of tests should be attempted only with great 

. ; ..|,tor i. ii f'>1h>vv5, several other types of instruments which 
i,- ^ M^v*-! icu ally ctre discussed. While the discussion will be in 
f ? ritii: trc-atment va^-iables, some of the techniques 

iiF< ission of questionnaires) may be useful and easily adapted 

i (fu^exi of ijieasuring student outromes» 
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CHAPTER VII 
MEASURING PROGRAM TREATMENTS 
Process or treatment evaluation Is Important because program managers need to 
know ^'why ' a program worked just as much as they need to know how successful 
It was. In general terms, accumulation of data required to measure project treat- 
ments is based or the same series of steps taken in measuring the etudnt outcomes* 
Specifically, it is first necessary to identify treatments and to formiilate a set of 
treatment questions. Second, it is necessary to specify those who will {srovide the 
information necessary to answer the treatment questions. Third, it is necessary 
to select or develop the instruments with which to obtain the information required 
to answer the questions. 

The first and most difficult step is to decide what strategies are being implemented 
that discriminate the career education effort from other facets of the school pro- 
gram. By definition, an innovative career education program will possess 
characteristics that differentiate it from more traditional educational practices. 
These characteristics are the ingredients in the program's ''recipe. " If the "recipe" 
can be explicitly formulated, then it is likely that it can ba replicated or improved. 
If it cannot. .*ven if a program is judged successful in terms of the student outcome 
clata. the staff will not know what dimensions of the program were responsible for 
success. 

Specifying the treatments and relating them to participant groups was accomplished 
in the course of completing the tables and matrix discussed in chapter IV. Formu* 
lating the evaluation questions based on the tables and matrix was discussed in 
chapter V. The preceeding chapter dealt with obtaining measures of student 
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outcomes. ThLsf t^haptiT discusses the identification of the sources from which 
treatment data niav be obtained and approaches to securing that information. 

Onc4- thf program tre atments have been identified and appropriate questions 
posed, thv nrxt strp is to identify the sources from whi^'h data will be 
collritt fi to ansvvrr thpsr questions. In specifying treatments and formulating 
questions, inevitably some consideration was siven to the sources from 
vvhirh thr answers were to come. In fact, Lf the questions were formulated 
orcciKrly, more precisely th-^nis often the case, this step will have been 
actroniplishi fl as an inte^ral part of formulating the question. It is included 
here simply to insure that explicit consideration is given to alternative 
sources which may be possible and to assist in the selection or development 
of ni€*asurfnient instruments. 

Gf'n< rallv. th<-r*- are three possible sources of data for any given treatment 
question. Data may be obtained singly or in combination from: 

• the providrr of the treatment: 

• the rr>cipirnt at the treatment; and 

• an observfr of the treatment. 

Thf first task in determining appropriate data sourceis) for answering a 
civrn treatment question is to prepare a simple list of the groups of 
individuals in each of the above three categories. For example, a question 
miiiht bo: How does the classroom experience of participating third graders 
diffrr from non-participating third graders along the dimensions of U) the 
amount of c lass time devoted to discussion of various jobs and their require- 
ments, ihS the number of field trips to see workers in their work setting, 

ERic m 
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and (c) the number of class visitors who have discussed their jobs? Given 
this question, possible sources of the data necessary could be ordered as 
follows: 

Providers: third grade teachers, school staff members who schedule 

field trips, and class visitors. 
Recipients: third grade students. 

Observers: school principal. Career Education coordinator. 
Once a list of possible sources has been made, those sources from whom It 
would be feasible to collect daU should be identified. The next task is to 
determine from which source the information is likely to be most accurate. 

Once a decision regarding the data source has bean made, it is necessary to 
identify the characteristics, if any, wh*ch differentiate the sources into impor- 
tant sub-groups. For example, it ms ' judged that the differences among 
teachers with respect to sex, age. number of training sessions attended, or 
attitude toward career education, might affect their responses to treatment 
questions. It may be of interest, for example, to relate selected teacher char- 
acteristics to the extent to which they utilize outside speakers or provide odier 
treatments. Gathering this descriptive data regarding respondents will require 
supplementing the instruments to be selected or developed in che instrumenta- 
tion step of the evaluation process with appropriate items. 
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The final lank in identifying appropriate data sources in to determine whether 
the method of collection should be direct or indirect. In other >vorda» must 
thf* individuals who will supply the data be asked in some manner for the 
information or does a system already exist which provides suitable informa- 
tion. If such a reporting system does exist, then the Instrument selection/ 

development step will consist of developing forms for recording data already 
available. If such a systcrm does not exist, then the Instrument task becomes 

som<*what more difficult. 

B. Instr um ontation 

Once the treatments are identified, the questions formulated, and the data 
sources identified, the next step is to select or design appropriate infitrumen- 
tation for collection of the data. Very rarely will the needed treatment 
instrumentation be commercially available or available from other career 
education programs because of the specificity and uniqueness of most 
treatments and program contexts. 

While instrumentation typically must be site specific, it is possible to 
identify various types or approaches to instrumentation. Among the possible 
tyoes of instruments which have been used in collecting treatment data are: 
m document review Summary forms; 

• questionnaires: 

• observation scales; 

• staff reporting forms; and 

• interview schedules. 



ERLC 



ERIC 



81 

Document Review Summary Forma : Data pertaining to some treatment 

questions may be included in documents regularly prepared by the school 

district for other purposes. For example, in some school systems It may 

be possible to assess the extent to which a career information center Is 

utilized by various groups of students by reviewing the records kept by the 

center sUff or school librarian. If the center has been in operation for 
several years and if comparable records exist, it miy be slble not only 

to assess the extent to which particular groups have used materials during 

the evaluation period, but also to assess the extent to which they had used 

materials in prior years. By the same token, it may be possible to obtain 

information on the number and type of field trips taken In some schools by a 

review of school transportation records (if walking trips are not a feasible 

option in that situation). In many situations counselor records may also be a 

potent source of information. 

As is true in development of all instruments, the key In developing document 
reporting forms is to know precisely what information Is desired. Presumably 
this will have been accomplished in the process of Identifying treatments and 
formulating evaluation questions. In addition to having a clear understanding 
of the information desired, it is wise to have a thorough knowledge of the 
documents which are to be reviewed prior to developing the forms. School 
files and other such documents are rarely designed to provide 
precisely the information desired by a program evaluation. As a result, 
it is necessary to develop summary forms which can obtain the maximum 
information available with the least amount of effort. 

.89 



While the feasibility of obtaining the needed data from particular documents 
should have bec-n a consideration In identifying the data sources » it may be 
necessary to reconsider the appropriateness of the decision once the form 
development process has begun. It is not unusual for a cursory review 
of documents to suggest that they contain information which xxpon 
closer analysis is found to be internally inconsistent or incomplete* It 
is also not unusual to discover that records do not report what they claim 
to be reporting. Frequently analysis shows that what api^ars on the records 
to be the number of students placed on jobs is in reality the number of 
referrals of students to potential employment. Another esnmple of records 
not reporting what it seems often occurs when seeking the number of students 
receiving two or more treatments. The records may appear to Indicate that 
200 students are irvolved in various program activities when In facta group cf 
50 students is receiving four different treatments. Further, It Is not unusual 
for records to contain estimated figures which for most purpose are quite 
reasonable. However, for purposes of evaluation such estimates may not 
reflect subtle but significant changes and so may prove to be misleading. 

In summary, while obtaining inforiiiation from existing documents may at 
first seem to be the preferred approach on the grounds of ea e of instrument 
construction and convenience to the school district, experience suggests 
thai this is not always the case. Before concluding that this approach will be 
followed and finali/iins the summary cuUection forms, it is necessary to go 
through a trial run. It may be possible that use of certain existing 
O uments (e.g., counselor records) will not be possible or that which appears 
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to be an ideal format from the viewpoint of later analysis may be unduly 
cumbersome from the standpoint of searching the documents and recording 
the data. In all cases it is helpful, and frequently enlightening, to determine 
who completes a given record, when, and as a result of what action. 

Questionnaires : Sine*. - all necessary information will be found in pre-exist- 
ing documents. It is virtually inevitable that to obtain some information ques- 
tionnaires wi?l need to be developed. It Is likely that separate questionnaires 
will be developed for teachers, counselors, students, administrative staff, and 
others who provide or observe treatments. As used here, a questionnaire is 
defined as an instrument which is given to an individual with the understanding 
that the recipient will carefully answer in writing the questions asked. 

Ideally, a questionnaire would go through the same rigorous process of develop- 
ment as a test. In practice, however, generally this is not possible. Neverthe- 
less, in the context of evaluating a career education program in a school 
district, the questionnaire will often be the most feasible method of accumulat- 
ing answers to specific questions or sets of relatea questions fron-? treatment 
providers, recipients, and observers. Below some guidelines for questionnaire 
development are provided. 

In developing a questionnaii e there £<re several basic roles which should be 
followed carefully. While they sound simple and have been repeated in the 
literature for years, they are violated frequently enough to warrent repetition. 
The p^-imary rules are as follows: 

• The questiong should be simp'r and precise . Perhaps the most common 
difficulty In question construction Is what should be two or more 
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qiifstion.s ar«' combirn-tl. F<ir oxamplf, if program personnel wish to 

rn, . rtain -i^. a nianv of th.- .stiidrnts in a ninth grade Fnfiliah ciasa that 

\v« nt nn a particular fit ld trip wrote an oasay about it, st'vcral short 

qiu'stioHH art' necessary. It is not enough to ask: "Did you write an 
. sHav on ynur field trip:> ■ The first question should be: "Did you go 

(jn a fifld trip to X " 

The questi ons should be und erstandable . The words and concepts should 
b" rt-iidily understood by the reader. Ambiguous phrases should not be 
UH, <1. For example, if the need is to know how much time a counselor 
spent = 1 «.areer counseling, it is crucial to be certain that all recipients 
will use the same definition of the concept of "career counseling." If 
there is doubt, the question should be broken into various components. 
The criteria for providing the response should be clear. At times there 
are two or more perfectly logical ways to answer a question. For 
xample, if the need is to know how many students were seen by a 
counselor once and how many were seen twice, it should be clear whether 
the number seen once is to be included in the number seen twice or 
v.hcther the two groups should be reported separately. 
The response should require the fewest w '>rds from the respondent 
possible . Idrally, questions and possible answers can be so well thought 
out that the respondent will simply need to place a check mark In a 
desiunated squar*-. This reduces responding and interpretation time. 
Oft. n. howevr, this is rot possible and open-ended questions are 
nrc-ssary. Wm-re this is the case, tentative categories of responses 
should be prepared. In some cases it may be concluded in advance that Aere 
will be so litti*. con.monalitv in responses that an item should be excluded. 
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• Catch-all response items 8uch as "other" ahoiild be avoided where 

possible. The existence of such Items suggests that the question was 
not sufficiently thought out or that the Information desired was not 
clear. It is not unusual for such Items to produce no helpfiil Information. 
There is also the danger that significant data will be lost bec&use of 
the unspeclflc response. 

In essence most questionnaires are collections of single questions put 
together such that they are all asked on the same form. The construction 
of the Items and the order In which they appear then are the two most 
Important factors. In Item construction the preceding rules should be 
carefully followed. It may be useful to review questionnaires f > .->m other 
projects to identify items which they found successful. In using Items 
developed elsewhere, consideration should, howe/er, be given to the unique 
characteristics of the local respondents. The interpretation given to certain 
words or phrases may vary from location to location. 

The primary consideration in ordering the Items or sets of Items are 
(a) their Influence on responses to subsequent questions, and (b) the 
attitude engendered in the respondents. In the first Instance, care must 
be given to the possibility that the answer to one question will f'lctate 
somehow the response to other questions when this Is not desired. In the 
second, care must be taken to insure that the time necessary to respond to • 
the first item will rot lead the respondent to discontinue completing the 
questionnaire. 
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Observation Scales t In some situations it may be appropriate to develop 
instruments to be used by observers of classrooms or other project 
activities* The advantage of such an approach is that it permits an objective 
outsider to observe administration of the treatment; for example, the 
behavior of the teacher and pupils and their interaction. These observations 
may verify teacher reporting forms and/or may permit identification of 
facets of the classroom treatment which are not acquired through other data 
collection techniques. 

The difficulties with this approach are two-fold. Firsts it is never possible 
to obtain as many observations a^ might be desired* Implementation Is time 
consuming in that a third party must observe the treatment at least once and 
Ideally several times. Second, if more than one observer is used, it is 
difficult, if not impossible, to be certain that all are using the same criteria 
in making their judgments. Observation of an event is subject to influences 
beyond the control of the observer and descriptions of the same event from 
the same person may change over time. 

If this approach is utilized, it is important to develop observation scales 
which require as little subjective judgment as possible. As with question- 
naires, the more often the observer can res|K>nd by the use of a check mark 
the better. Second, it is important » if several observers are used that 
an assessment of inter-rater reliability be maHe, This simply means that 
several observers should view the same event and record their findings. 
The extent to which their reports are the same is the extent to which 
ERXC*li^hlllty among raters has been achieved. 
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Staff Reporting Forms : The purpose of a staff refwrting form is to elicit 
from the direct provider of a treatment (e,g#i the teacher), or indirect 
provider (<'«r«9 coordinator), a description of relevant treatments on a 
continuous basis. Two distinct uses can be made of the results of this 
reporting process. First, the data provided can be used by administrators 
to monitor the activities of the program and to take corrective action as 
necessary. For example, if the number of field trips taken by teachers was 
reported monthly and if by Christmas some teachers had taken their classes 
on no trips, the career education coordinator could try to Intervene, 

The second use of a reporting form is to provide a summary record at the 
end of the year of the treatments which were administered to various groups 
of students* While this Information might be secured through an interview 
or questionnaire given to staff at the end ot the year, it is likely that the 
information will be more precise and accurate If the staff Is reporting these 
events on an ongoing basis. 

Given that a purpose of the evaluation is to assist local efforts Improve, the 
two uses of the forms are quite compatible. As long as a record Is made of 
the results of the interventions made by coordinating staff during the course 
of the year, using the forms for monitoring purposes will not affect the 
ability to use them at the end of the year in relating treatments to outcomes. 
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Given th<? advantages to program manaRoment that utlli?'.atLon of this approach 
provi lrs, it la rocommrndcd for use to the extent feasible for any project. 
There are certain constraints on extensive use, however, and it should be 
used as a supplementary data collection device. 

The difficulties in adopting the staff reporting form strategy extensively 
are of three types. First there is the problem of precisely identifying the 
information which is to be reported and the development of forms which 
capture the information in a consistent manner across all staff members. 
Since this Is the same problem confronting developers of all types of 
instruments, it is not a sufficient reason to abandon this approach to 
data collection. 

Second, there is the problem of accurate reporting of Information. Since 
staff presumably will know that a use of the form is to monitor their 
activities, they may be more prone to provide less accurate Information 
than on other instruments perceived to have less Immediate impact. In 
part, this can be resolved through providing the staff with an understanding 
that what Is sought is accurate information and that punatlve judgments will 
not he based on the Information provided. In addition, simple procedures 
can be developed to verify the reporting information. These might rely on 
v(.niparin« the self reported int'ormatior with school records indicating some 
<,{ thf sam.' areas. For example, field trips reported could periodically be 
cross checked against school transportation records. Class observation is 
another metho*! which could provide a periodic cross check on the results 



of the self- reporting forma* Finally^ students or others involved In the 
process could periodically be asked questions which would provide 
verification. 

The third and most significant constraint is the Imposition such systems 
make on staff time. If the system requires considerable time for the staff 
member to complete, there may be political or accuracy costs to pay. 
At the one extreme, teachers may simply refuse to complete the forms 
and at the other they may complete them carelessly and Inaccurately. 
It Is Important, therefore, to develop forms which are quickly and easily 
completed and for which the staff can see a useful purpose. A number of 
career education projects have had success in utilizing this method in 
collecting a major portion of the treatment data ai;a have developed forms 
which could be of use. 

In developing staff reporting forms or procedures, most. If not all, staff 
objections and Inherent dlfHcultles mentioned above can be overcome If the 
reporting form is a natural by-product of normal action Instead of something 
imposed in addition to normal procedures. If, for es^mple, counselors 
write a simple referral sheet when referring students to jobs, It is better 
to have a carbon copy of '/lat referral serve as the repe ating form than to 
have a separate summary form for the counselor to fill out. Summation 
then becomes a routine task which can be performed by support ««taff on a 
regular basis. 



Interview Schedules: The Interview schedule or interview guide is a method 
of insuring that information wliich Is best obtained orally is sought and 
recorded in a standard format. The interview schedule may be use d to 
supplement any of the other methods of data collection either to verify the 
information reported or to add to it. For some purpojea it may also be the 
only source of information. 

Such schedules are either administered directly in a face to face Interview 
situation or indirectly by telephone. In the first case, the format may either 
be quite tight, permitting little or no deviation from what is essentially a 
script for the interviewer, or may be quite loose permitting a conversational 
atmosphere, with a high probability that the precise order of the questions 
io b e asked will not be as they appear on the guide. 

The appropriate format to be adopted will depend on the use to which the 
information is to be put and the number and status of the Interviewees. If 
there is a single interviewee (e.g., the school superintendent), then both 
methodological and pragmatic reasons suggest the looser format. The 
answers will not be directly compared with those of a similar Individual 
and the superintendent is likely to resist a rigid interview format. On the 
other hand, if the purpose is to elicit standard responses from a group of 
teachers or students, then a more structured approach may be necessary. 

\ 
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As the sole source of a type of treatment data, there are three situations 
where this approach to data collection may be preferred* First, when the 
respondents are unlikely to resj^^nd to any other method^ this obviously is 
the preferable approach. This may occur when information from employers 
providing training to high school students Is necessary. Often there is 
greater possibility of obtaining a brief interview with an employer who 
supervises a student in a cooperative education program than there Is of 
securing a written response to a questionnaire* 

Second, when the respondents are expected to have difficulty with a written 
form based on linguistic difficulties. This may be due to their fluency in 
written English or the Inherent ambiguity of some terms associated with the 
educational program* In both cases it may be judged essential' that an 
Interviewer have the opportunity to be certain tnat the respondent fully 
understands the questions asked. 

Third, when the purpose is to explore an area wltii the respondent* In some 
cases It may not be possible to be precise with regard to questions in some 
areas. At times, what is sought is the outcome of following a chain starting, 
from a precisely formulated question which could be answered in a multiplicity 
of ways. It may be, for example, that when the list of treatment questions 
was developrd it was Impossible to define precisely what happened In some 
locations. For example, it may not be possible to go beyond the geni?ral 
level of ^'distributive education grocery sales*^ at the start of the design 
process^ In such a case it may be necessary to Interview the teacher, 
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sever^tl ^;tudentSt and employers with regard to identifying the specific nature 
of the treatn^ent« Prom the results of these open-ended interviews would 
emerge the needed description of the finer points of the treatment category. 

Regardless of the approach or combination of approaches used to obtain informs^ 
tion regarding a specific research question^ it is clear that instnunent develop- 
ment requires ^^areful, time consuming labor. In all cases it requires pradsioa 
with regard to the information sought and an understanding of the uses to which 
the information is to be put. It also requires an assessment of the practicalltiesl 
of data collection such as ease of administration^ and the time required. Most 
important, it requires attention to the issue of obtaining valid and reliable 
responses; the various cautions, guidelines, and suggestions contained in this 
chapter are designed to help in this regard. 

Another factor to be considered is the ease of quantifying the treatment 
variables. Some would be relatively simple^ such as the number of field trips 
or the number of different occupations represented by the outside visitors to a 
classroom. Other potentially influential factors, such as the emphasis placed 
by the teacher on career decision-making, may lend themselves only to very 
coarse quantitative categories, e.g., '^much emphasis,^' "little empl^asis, " cr 
'^^one at all. In the latter instances, it is necessary to have more than one 
jtidge assign these '^quantitative'' labels to the specific data, and to utilize the 
pooled judgments as the measure* Regardless of the method used for quantifica- 
tion, or the precision of measurement which may be possible^ it is essential 

that each variable be quantified in order to be able to assess the relationship 
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between the treatment variable and its relevant outcome meaeureCs). Such 
assessment Is only possible if outcome data can be grouped into at least two 
categories pertaining to treatment* such as '^treated'* and *'not treated* " or 
'liigh exposure to treatment" and "low exposure to treatment. " 
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CHAPTER Vm 
SAMPLING PLAN 

The next atep in the process is to specify the sampling plan. In an educational set» 
ting there are often many restrictions on how samples can be selected; this in turn 
will have important implicatiwis for the data analysis techniques which can or cannot 
be utilised (see Data Analysis Plan). One of the principal sampling restrictions is 
that it is often difficult to select a random number of students from a school or from 
pre-selected classrooms within a school. Often it is necessary to randomly select 
schools or intact classrooms within schools. If the evaluator decides to randomly 
select classrooms rather than students p then his unit of evaluation becomes the class- 
room, rather than the student If f<mr classrooms of 25 student -, each are selected, 
the size of the sample will be four, rather than 100. This fact is . ^Un overlooked by 
researchers who use the number of students as the sample siae wispite the fact that 
the logic of their statistical tests may require that the unit in the sample be selected 
<m a random basis. A more complete discussion of the appropriate eKperimental 
unit in statistical analysis can be found in Julian Stanley's (ed, ) Improving^ Experi* 
mental Design and Statistical Analysis , Our purpose here is to point out that the unit 
of analysis will vary depending on selection considerations. 

Assuming that an adequate sample can be drawn, considering the design and unit of 
analysis, we may proceed to the next and most frequently asked questicm: ''How big 
a sample must I take?" Since we always obtain more accuracy and more power in a 
test by increased sample 3ise, a larger sample size would be better than a smaller 
one. In other words, with all other factors held constant, the larger the sample, ^he 
smaller the sampling error. The uinderlying question is '*How much of a sampling 
error are we willing to put up with?^^ The answer depends upon: a) the consequences 
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of reaching wrong conclusions; and b) the cost of reducing the likelihood of being 
wron^. 

The decisions involved in answering the above questions cannot be made on the basis 
of statisti' al or methodological considerations alone, since asarssment of the con- 
sequences of error of given magtntudes and cost factors are essentially admlnUtra- 
tive considerations. Further, they depend upon the distribution of the variables in 
the population, and this is generally not known. As a reasonable approach to these 
problems, most studies of this type develop sampling plans which call for samples of 
approximately (with a minimum of 30 If the group is relatively small) where 
the population numbers up to 5,000. For larger popuUtions, over 5,300, a sample 
si^e of approximately 2«;, is generally considered adequate. In other words, sample 
sizes of between 30 and 250 where the population ranges up to 5,000 and of at teast 
100 for populations in excess of 5,000 are generally viewed as being stable enough 
to insure a sviff iciently low error of estimate with a reasonable i:xpendlture of 
time and money. Of course, these sample sizes may vary in accordance with 
the specifics of the research design. Either the 90% or the 95% confidence level 
is utilized to judge the significance of the findings. 

The preceding rules of thumb with regard to sample size apply to each "homo- 
geneous" group, e.g., students in the 9th grade who comprise treatment group 
X. or students in the 9th grade who constitute a "no treatment" group. It 
also refers to samples in which the students in the sample Lave been randomly 
selected. Where intact classrooms have been selected, i.e., where cUssrooms 
rather than students constitute the unit of analysis, it is necessary to select 
as ntany classrooms as possible to maximize the stability of the findings and 
to maximize the sensitivity of the statistical techniques which are applicable 
to sm*.ll samples, i.e., samples between 10 and 20. 

Sampling, then, requires a knowledge and implementation of procedures which 
allow for an unbiased selection of u small group that is characteristic of the larger 
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popuUtion such that estimates based on this smaUer group can be generalised to 0ifi 
larger group from which it has been drawn. Of practical concern is the capabUity of 
drawing the sample according to the research design, the siac of the sample, and an 
understanding of the statistical techniques that can or musi be employed. It is not 
xinlikely that the evaluator will find himself in a position of not having enough sub- 
jects, or resources, to investigate the large number of dimensions to be studied 
simultaneously. In this case, he may modify his design to accommodate the re- 
sources avaUable. Matrix sampling, for example, might be explored in situations 
wi»re tliere are relatively large n imbers of students, the number of instrttments 
(or items) to be used is quite large, and the time for testing is minimal. * 

In summation, program personnel considering sampling issues should keep the 

foUcwing points in mind: 

• Sampling is not an issue when aU of the treatment population is tested. 

• A census i« the moat accurate measure. Samples are taken when a 
census is not possible or practical. The Urger the sample the better 
the measure. 

• When the number to be tested is less than a census, random selection is 
necessary if the res\ilts are to be generalised. 

, Confidence levels and therefore sample slaes can vary based upon 

practical or administrative c<mside rations. 

• Asa rule, for treatment populations of 5. 000 or under a sample si«e of 
5% or 30 participants, whichever is the larger, is considered an 

*Note: In general, the design of the sample is a very sophisticated problem requir- 
ing the expertise of a statistician. If no member of your staff has sufficient back- 
ground in this area, an outside statistician should be engaged as an advisor. 

X04 



appropriate or safe sample. 

The SO or 5% rule applies for each treatment group. 
If classrooms are selected as the sample unit, it is likely that more 
students will have to be tested to insure a nafe sample than if the 
student is the sample imit. 
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CHAPTER IX 
DATA ANALYSIS PLAN 

The speciHcation of an analysis plan usually refers to the selection of the 
statistical method to be employed in analysing the data collected. Some 
statistical methods reqtdre that different kinds of data be collected than do 
others. For this reason, if the statistical method to be employed during 
analysis is specified prior to the collection of the data, it is necessary to 
consider the type of data required by that method prior to development of the 
sampling plan. In other words, the sampling plan would need to be designed 
so as to fulfill the requirements of the particular method chosen. As a 
practical matter the specitication of the method to be used prior to development 
of the sampling plan serves primarily as a guide in developing the sampling 
plan. 

Usually, however, the sampling plan will be developed around other con- 
siderations, such as time, money, practicality and av&llability of data 
processing resources. Most programs cannot afford the luxury of designing 
around analysis technique prerequisites. Therefore, the selection of the 
specific statistical methods to be employed is frequently based upon findings 
made after the data collection has been completed. 
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At this point in the evaluation process what is recommended is: (a) to specify 
the data processing requirements implicit in decisions regarding instrumentation 
and sampling, and (b) to prepare for the practical implications of the probable 
final analysis plan. In essence, this simply means to: 

• estimate the time, personnel and equipmeitf necessary; and 

• arrange for its availability. 

The first stage of data analysis begins immediately upon the completion of the 
collection of data. It involves making frequency counts and computing other 
descriptive statistics on all variables for each treatment group. Depending 
on instrumentation, getting to this point may or may not be a time consuming 
endeavor. This process should make poesible the identification of non- 
productive variables, that is, variables with little or no response range. 
Since these cannot be of value in the analysis, they should be eliminated from 
further consideration. 

Tht! material w!-«ch follows provides an overview of what statistical technicpes 
are appropriate under what conditions. This is intended to serve as general 
background information for the next step of the analysis plan which is 
specifying the statistical techniques to be utilized* 
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SpactiVlng tha Data Analysis Techniqueii 

Every statistical procedure for analysing data is designed to ^swer a 
specific type of analytic question and is based upon certain assumptions 
regarding the nature of the data and the study design. The use of procedures 
designed for answering a different type of question than ti&ose discussed in these 
guidelines would be wasteful, while the use of procedures having data and/or 
design requirements which cannot be met would be erroneous and misleading. 
For these reasons, this sectton will focus only on those procedures which are 
apt to be appropriate; fur^er, since the evaluation is oriented chiefly toward 
local use of the findings, it will focus on those procedures which yield 
findings that are relatively easy to interpret. TUs focus, however, is not 
intended to imply that expert statistical consultation may be dispensed with, 
whether at the design, data analysis, or data interpretation phase of the 
study. 

Statistical techniques may be grouped into two large categories; parametric 
and non- parametric. The parametric techniques, which are more frequently 
used than the non- parametric, are based upon different assumptions, the 
most important of which is that the sample or samples must have been 
randomly drawn from one or more normally distributed populations. The 
advantage of these techniques is that Uiey are more "sensitive" in detecting 
statistically significant findings (this is called the "power" of a test in 
technical terminology), providing that the sample is sufficiently large. For 
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small samples, and for larger samples which are not drawn randomly from 
normally distributed populations, it is necessary to utilize the less sensitive non- 
parametric techniques. By way of example, it is anticipated that the non-para- 
metric will normally be used when dealing with data from the elements ry grades, 
where classrooms, rather than students, are apt to be the unit of analysis. la 
this instance, the number of classrooms is likely to be small (e, g. , 10 or 20) 
and the assumption of a normal distribution of classroom means on the outcome 
measures would be difficult to defend. 

To answer the specific study questions formulated as Step 3 in the evaluation 

process, it is necessary to determines (a) the magnitude cf the pre -post change in 

test scores for each treatment group of students or classroons, and (b) to asjess 

the significance of the differences in change among the several treatment 

groups. This process is shown schematically in the diagram below: 

Scores on Scores on 

Pre- Test Post-Test Difference 

Treatment Group A B B-A 

Comparison Group C D D-C 

Where the sampling plan justifies the use of parametric techniques, and where 

there are only two groups, the statistical technique known as the t-test for 

correlated measures would be applicable. If more than two groups are 

involved, and it can be assumed that students have been assigned to groups 

on a random basis, the basic analysis would be carried out by means of 

the analysis of covariance technique. Information concerning the pre-test * 
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scores alone or the post- test scores alone vo uld be obtained by the means of 
the analysis of variance technique. Where non- parametric techniques muct 
be useo, the Mann- Whitney U Test and MacNemar's Test for Change would be 
apolica.ble if two groups are involved, and the Krushal- Wallis One-Mlty 
Analysis of Variance and the Chi-Square Test iray be utilized to assess the 
significance of the inter- group differences. 

Either within groups or across groups, correlational methods may be utilized 
to determine the "strength" of the relationship between treatr ^t variables 
and outcome variables. For relatively large samples, the Pearson product- 
moment correlation would be applicable if there is a fairly large variation in 
the tre-.tment variable; for small samples, the Spearman Rank Order 
Correlati^ n wcxild be applicable. 

The potentially applicable techniques are described briefly below, and 
references to statistics books are cited for the interest of those who wish to 
investigate the technical aspects further.* The descriptions are provided 
first for parametric techniques and then for the non- parametric. They a^e 
arranged and numbered so that corresponding numbers indicate equivalence 
with regard to the type of study question for which the procedures are 
applicable. For example, the t-test (parametric technique 1) is equivalent 
to the non-parametric "Mann-Whitney U" (non-parametric technique 1). 



* The sections on specific statistical techniques and the list of references is 
qnoted, with pet-mission, from Stenner, A.J. , and Webste*-, w. J. , 
Educational Program Audit . , Arlington, Va, , IBEX, Inc. , 1971. 
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A. Parametric Techniques 
1. The t^teflt 

"The t-test is a statistical model that is desigbed to investigate 
several types of questions. The one- sample t-test determines, 
within specified levels of probability, whether or not the 
population from which the sample was drawn has a given mean. 
The two- sample t-test determines, within specified limits of 
probability, whether or not the populations fium which the 
samples were drawn have the same mean. Different models 
are used, depending upon the type of sample data under 
analysis. In yielding a probability statement of the differences 
between group means, the nusdel considers mean differences, 
sample variability, sample size, and whether the data are 
correlated or independent. (Blommers and Llndqulst, I960, 
chap. 12; Edwards, 1957, chaps. 13 and 14; Games and Klare, 
1967. chaps. 11 and 12; Hays, 1963, chap. 10; Kendally and 
Stuart, 1967, chap. 11; Popham, 1967, chaps. 10 and 11; Wert, 
Neidt, and Ahmann, 1954, chap. 8. )" 

2. The Analysis of Variance 

"When two or more groups or samples are available, the 
analysis of variance is a mo'^ei ' sed to test for differences, 
within specified limits of probability, between the means 
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of those groups. The procedure enables one to analyze 
variances in such a manner that conclusions can be drawn 
about means. In the one-way situation (only one independent 
variable), the procedure is merely an extension o' "-e t-test 
to situations where there are more than two groups. Higher 
order analysis of variance is used to ascertain, within 
specified limits of probability, the effects of two or tnore 
independent variables on a dependent variable. Utilizing 
higher order analysis of variance, the researcher can test 
main effects and Interaction effects. In higher order analysis 
of variance, the researcher must concern himself with whether 
he is deaUng with a fixed, mixed, or random model, since the 
type of model used will, in the presence of a significant 
interaction effect, make a difference in the error term used. 
Among the types of designs available with this technique are 
factorial designs, repeated measures designs, Latin Square 
designs, and numerous permutations and combinations of 
the various techniques. (Cochran and Cos, 1957; Cox, 1958; 
Edwards, 1957, chaps. 10 and 17; Edwards, I960, chaps 9 
through 13; Guenther, 1964; Hays, 1963, chaps. 12 through 14; 
Kendall and Stuart, chaps. 35 through 38; Lindquist 1953, 
Scheffe, 1959, Walker and Lev, 1953, chap. 9; Wert. Neidt, 
and Ahmann. 1954, chaps. 10 through 12; Winer, 1962. )" 



106 

The Analysis of Covariance 

"When two or more groups or samples are available, the analysis 
of covariance is a model used to test for differences, within 
specified levels of probability, among the means of those 
groups, after initial compensation for differences among grouos 
with respect to one or more control variables. Analysis of co- 
variance is not a method to adjust for lack of random sampling; 
it is rather a technique that can be used to increase the precision 
of one's experiment if the control variable (s) selected is 
highly linearly corre lated with the dependent variable. 
Analysis of covariance can be vs ed in simple- classification or 
multiple- classi^cation forms and, when the assumptions are 
met. is a more powerful technique than analysis of variance 
because it will provide a reduced error term. (Edwards, I960, 
chap. 16; Guenther, 1964, chap. 6; Wert, Neidt, Ahmann, 1954, 
chap. 18; Winer, 1962, chap. 11.)" 

Correlation Analysis 

"When the words correlational analysis appear in the Metric- 
Metric cells of the taxonomy, they refer to the use of the 
Pearson product- moment correlation coefficient. Basically, 
correlational analysis provides the rese trcher with procedures 
for quantifying the measured relationships between two or 
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more variables. The size of the product* moment correlation 
coefficient varies from *hOO to +L 00* thus providing the 
evaluator with an estimate of the size and direction of a 
given relationship. 

"If the evaluator is interested in studying the relationship 
between two variables he could use the product- moment 
correlation coefficient. If he is interested in the relationship 
between one variable and a combination of two or more other 
variables considered simultaneously, he coxild use a multiple 
product- moment correlation coefficient. Multiple correlation 
provides an index of the relationship between a single metric 
variable and a composite. It is often evident that a relation^ 
ship of some Interest may be explained^ at least io some extent* 
on the basis of correlations with a third variable or composite. 
In such a ca^e, the evaluator may wish to determine the degree 
to which the two variables of primary interest are related 
beyond the relation implied by correlations with a third variable. 
The partial correlation between the v^ariables of primary 
interest provides a measure of their relationship independent 
of some other variate« (Blalock^ I960, chaps. 17 through 19; 
Blalock, 1964; Edwards, 1957, chaps. 8 through 10; Guilford, 
1965, chaps. 6, 14 and 16; Kendall and Stuart, 1967, chaps. 16, 
26 and 27; McNemar, 1962, chapa 10 through 12; Walker and 
Lev, 1953, chap. 11; Zctterberg, 1963. 
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Regregsion Analysis 

"Regression analysis is usually presented in the educational 
and psychological literature as a model for making prediction 
on a given criterion from one or a set of predictors. The 
yield of the approach is a simple or multiple correlation co- 
efficient and a regression equation composed of a set of 
weights that can be used to optimise prediction. (Bottenberg 
and Ward, 1963; Draper and Smith, 1966; DuBols, 1957; 
Johnston, I960; Kelley, Beggs, McNeil, Eichelberger, and 
Lyon. 1969; Kendall and Stuart, 1967, chap. 28). " 
Discriminant Analysis 

"Discriminant analysis represents an extension of regression 
analysis to the case where the criterion variable is descrete 
rather than continuous. The model is generally used for 
classification purposes. In essence, it provides an estimate 
of the positiai of an individual, based on specified information 
on that individual, on a line that best separates two or more 
classes or groups. Since It is often the case that one best 
line may not exhaust the power of a given group of measures 
for discriminating among groups, additional discriminant 
functions (to the lesser of the number of groups minus one, or 
the number of measures) may be fitted. Thus, the major 
purpose of discriminant analysis is to determine whether 
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discrimination among grwips on the basis of a specified set 
of variables is possible or not, and then to reduce the siae of 
the predictor space without substantial loss of information. 
Discrimination analysis itself does not define the regions of 
classification; however, approximate tests of the statistical 
significance of the separati<m of groups on a particular discri- 
minate function are available, and the relative contributiojs 

of original variables to a discrimant function can be shown. 

(Anderson, 1958, chap. 6; Cooley and I-ohnes, 1962, chap. 6 

Kendall and Stuart, 1967, chap. 44j Morrison 1967, chap. 4; 

Rao, 1952, chap. 8.)" 

Non-Parametric Techniques 

1. M»«n-Whltnev U 

'•This non -parametric technique haj? been developed indepen- 
dently by a great number of writers to test, within specified 
levels of probability, whether the populations fror^i which two 
independent samples have been drawn have the same dlstri - 
bution. It is one of the most powerful non -parametric tech- 
niques and is a useful alternative to the parametric t-test. 
(Bradley, i960, chap. 5; Hays, 1963, chap. 18; KendaU 
and Stuart, 1967, chap. 31; Siegel, 1956, chap. 9j Walsh E 
1965, chap. 2.)" 
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2« Kruskal-Wallia One «W ay Analysis of Variance 

^^The Kruskal-Wallis one-way analysis of variance is useful 
for determining, within specified limits of probability, whether 
or not the populations from which two, or more independent 
samples were drawn share certain specified characteristics. 
It is an extremely powerful non-parametric technique ar^d a 
useful alternative to the parametric one-way analysis of vari- 
ance. Procedures for post-hoc analysis ^ve been developed. 
(Bradley, I960, chap. 12; Hays, 1963, chap. 18; Kendall 
and Stuart, 1967, 1967, chap. 31; Siegel, 1956, chap. 8; 
Walsh II, 1965, chap. 4. 
(3. No equivalent technique) 

4. Spearman Rank Order Correlation Coefficient (Rho) 

^'This statistic is a measure of association betwen ranks. 
It can be used for small or large samples, yields a coefficient 
between •I. 00 and +1. 00, and can be tested for statistical 
significance^ It is an extremely powerful non -parametric 
technique for use with two ordinal variables. (Blalock, I960, 
chap. 6; Hays, 1963, chap. 18; Kendall and Stuart, 1967, 
chap. 31; McNemar, 1955, chap. 12; Siegel, 1956, chap. 9; 
Walsh I, 1962, chap. 5. y 
(5. No equivalent technique) 
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( 6. No equivalent technique) 

7. Chi -Square 

"This is an extremely flexible and often used non-parametric 
statistical technique. It can be used to test the quality of 
population proportions (Chi -square test for homogeneity) 
or for statistical independence aimcmg related popidations 
(Chi-square test for Independence). Chi-square can be applied 
to more than two populations and more than a dichotomiza- 
tion of the variable or variables of interest. This is the only 
practical non-parametric technique in which it is possible to 
have both more than two samples and data which fall into more 
than two categories in the same design. Post -hoc tests are 
available for both Chi-square for homogeneity and for indepen- 
dence. (Bradley, I960, chap. 3j Hays, 1963, chap. 17; 
KendaU and Stuart, 1967, chap. 33; Siegel, 1956, chap. 8; 
Walsh II, 1965. chap. 5.)" 

8. Mc Nemar's Teat for Change _ 

"The McNemarU test for change determines, within specified 
levels of probability, whether or not observed changes in two 
related samples on the variable or variables of interest are 
greater than would be expected by chance. (Bradley, i960, 
chap. 4; Siegel, 1956, chap. 4.)" 
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9. Sign Test 

"Thf si^n tcHl is .tpplicablo to situations in whith tlitrrc 
are two related samples and the evaluator wishes to deter- 
mine, within specified levels of probability, whether thf> 
populations from which the samples were drawn share 
certain specified characteristics. The test focuses on 
the direction of differences between each pair of scores and 
is generally used to attempt to detect a shift in location. 
Probabilities arc determined by reference to the binomial 
distributicm. The power-efficiency of the sl^n test is high 
for small samples (n=6) but declines as sample sise increases. 
(Bradley, I960, chap. 2; Hays, 1963, chap. 18, Kendall and 
Stuart, 1967, chap. 52; Siegel, 1956, chap. 9*, Walsh I, 
1962, chap. 7.)" 

It should be stressed at this point that it is not the intention of this section to 
provide the reader with a "cookbook" set of instructions regarding data analysis. 
On the contrary, it is intended to make the point that statistical consultation is 
considered to be mandatory, both in formulating and in implementing the data 
analysis plan. 
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nATA ( Oi.i.rc T IO \' PlJiS 
The key to successhii data . 4)ll.'t tinji i-^ a sst ll tii0 4i;ht out plan that identifies 
the logistical activities in ativanci*, provides a basis for logically selecting 
the peraonnel rcquirod to collca whc data, and specifies time guides for 
judging progress. Tablo X-A ilhititrutcs a nine-stop plan which can aid in 
deaigning a data collection effort that is appropriate for your Career Education 
evaluation study. Tables X-B, X-C, and X-Darc provided at the end of this 
chapter as aids in developing tt;e plan. 

The first t^tcp in designing a <iata collet tion plan is to develop a list of all 
the tasks involved (the tasks may be listed in column 1 of Table X-D). In 
preparing the list, think of a task as an)thing that someone will have to do 
to carry out the data i ollcction proi-ess. Do not overlook simple activities such 
as the reproduction of materials. Often these small tasks cause major 
problems for the entire evaluation effort. In recording all the tasks, list them 
in sequential order so that nothing will be overlooked. When finished, carefully 
examine the list to sec that nothing is fnissing and that tasks flow logically from 
one to the other. 13c sure that you are not expecting some activity to occur 
without having a task somewhere along the line to carry out tfiat activity. 

The second step is to establish completioi. or target dates for each of the tasks, 
(the conipletioi lates may be entered in » ul. iirn Z of Table X-B). These dates 
should be realistic and, if possib!.*, provide some flexibility or cushion to allow 
for unexpected delavs lu- sure to take into a* t ouni holidays, vacations, school 

ERJC;ivitse8. etc.. when assigning complct*»5|atos to the tasks. With step two 
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completed you should know what has to be done and when* 

Step three involves the identification of the skills needed to complete each of 
the tasks* (These may be recorded in column 2 of Table X-C*) One example 
of such a skill might be a knowledge of test administration. Another might 
be in the area of telephone interview!]!^ • 

The fourth step involves identification of the personnel available to carry out 
the data collection (These may be recorded in column 1 of Table X-D.) 
You will need to know the constraints on the use of various personnel* Some 
will have only certain hours or days available to work on the data collection. 
Another consideration will be the cost. Some personnel will have to be paid 
out of project funds* In these cases, the budget will have to be considered 
in deciding who to use. In addition, an assessment of the skills possessed by 
the available personnel should be made* (Constraints on the use of personnel 
may be recorded in column 2 and notation of their relevant skills in column 
3 of Table X-D. ) 

The fifth step involves matching the personnel and tasks using the informa- 
tion you have developed in the previous steps* In some cases you may 
want to assign prime responsibility and also additional personnel for support 
activities (tnesc assignments may be recorded in columns 3 and 4 of Table 
X-B)* 

The sixth step involves the determination of the training needs of the 
personnel you have decided to use* This can be done simply by examining 
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the required skills as you have outlined them In step three (Table X-C, 
column 2) and matching them to the skills of the personnel you have selected. 
Discrepancies between the two will have to be resolved by training the 
personnel to meet the project needs. (The discrepancies may be recorded 
in column 3, Table X-C.) 

The seventh step requires the development of a training plan for providing 
the identified skills in the previous step. 

The eighth step is the implementation of the training plan. It is not 
appropriate to include here detailed information on carrying out the training. 
However, the following example may be helpful. In cases where you are 
using instrtiments with which personnel are unfamiliar, there should be a 
training session to familiarize the staff with the instrument. An example of 
an outline for such a training session is: 

A. Distribute samples of the test and answer sheets to the trainees. 

B. Discuss the test, explaining all relevant aspects. 

C Give the trainees copies of blank answer sheets or use a device 
for projecting the information for all to see. Discuss the 
necessary identification items on the answer sheet and the methods 
by which the item rejponses are to be indicated. 

D. Administer all or part of the test to the trainees. 

£. Discuss the results and the interpretation of the instrtiments. 

F. Clarify any queations the trainees have relative to the instrument. 
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G. Discuss the advantages and disadvantages of this particular 

instrument and any particular problem the trainee might have 
in administering the instrument. 

Having completed the above eight steps, you have almost completed your data 
collection plan. The plan should bethought of as flexible and as a guide. When 
changes must occur, do not abandon the entire plan but change it to meet the 
new needs. A good data collection plan can be a real time saver but it almost 
always goes through a number of revisions. 



As a last step you should evaluate the data collection plan after data 
collection has been completed and note any shortcomings that should be avoided 
during subsequent evaluation efforts. 

In some instances all or part of the data collection will be completed by an 
outside contractor. When this occurs, steps 4 through 8 will be completed by 
the contractor in the proposal submitted to the program. In reviewing any 
such proposals the program staff should note the degree to which the points 
raised in these steps are considered and realistic. 

When outside contractors are utilized the program will have to formalize the 
cost parameters. If no contractor is able to bid on the work to be done 
within the cost guidelines, either the work expected will have to be revised or 
the budget increased. 
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Table X-C 
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CHAPTER XI 

FINALIZE THE DESIGN AND PREPARE A REPORTING PI^N 



On^B the evaluation questions have been selected, instruments chosen or developed, 
analysis plans made, and data collection procedures established, it is necessary to 
review the evaluation design in its entirety in terms ol costs and limitatioxis. Once 
this is complete and any modificatiors n^ade as a result, the fixial step in the design 
process is to prepare a plan for reporting the results of the evaluation. 

A. Design Review 

Throughout the process of specifyteg questions and instruments, and settling 
on appropriate sampling and analysis strategies, a series of decisions have 
been made which have possibly (a) redefined somewhat the original conception 
of the final products of the evaluation, and (b) affected the expected cost of 
conducting the evaluation. Before finalising the design, it is necessary to 
review this series of decisions to insure that the results are acceptable. 
Systematic recording of basic decisions on the Evaluation Design Worksheet 
presented at the conclusion of chapter IZ will greatly facilitate this process. 

in most situations the final arbiter of design considerations will be the overall 
cost of the effort While in theory it might be preferable to decide what needs 
to be done and then allocate the resources necessary to accomplish the task, 
in reality this is rarely, if ever, possible. Typically a ceiling cost of the 
evaluation was sec well before the details of the design were specified and 
the issue at this point is to determine whether the ceiling was or was not ex- 
ceeded. If it was not, then the review may fmrus entirely on substantive 
matters. If the absolute ceiling was exceeded, then compromises regarding 
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the substantive products of the evaluation must be made. This should be done 
at this point in the design process. 

Once an assessment of the relationship between the cost of the evaluation to 
the resources available to conduct it have been made, the next step is to 
conduct a substantive review of the design. This review shmild begin with 
a reconsideration of the general questions which the various parties interested 
in the career education effort are likely to find of use and a determination of 
whether the design as constituted will provide the a^ropriate answers. It 
may be that for most i^ople the design will yield direct answers while for 
others the information produced will be indirect or inferential at best. 

In the likely event that certain questions are not directly answered, a deter- 
mination must be made regarding the possibility, given the nature of the 
career education effort in the district and methodological considerations, of 
providing a direct response. In cases where this is not possible, then this 
should be stated explicitly. On the other hand, if a direct response was 
theoretically possible and a decision regarding instruments or sampling was 
made l>ased on the time required to obtain the data or the availability or cost 
of instruments, then this too should be made explicit at this point. 

If the assessment of required versus available cost of the evaluation indicates 
that more funds are available than are required to toplement the design, then 
consideration should be given to refining the effort to answer the indirectly 
answered questions, or to addressing additional questions. On the other 
hand, if the cost requirements of the design exceed the available ftmds, then 
ERICipromises must be made, tl^gic|j|g^ these will either be (a) to eliminate 



"7 

altogether one or more questions, (b).to reduce the amount e£fort expended 
on answering certain questions. For example, at this point it might be wise 
to reconsider the pros and cons of utilising matrix sampling. 

Once the review is complete, the assumptions of the design and its limits and 
strengths should be made explicit, i. e. , a final version of the design work- 
sheet prepared. This will provide the basis for assessing the extent to which 
implementation has been successful and for informing interested parties 
(e. g. , the school board, teachers, etc. ) what the expected results are to 
be. In addition, it provides the basis for completing the final step in the 

design process: preparing the reporting plan. 
Reporting Plan 

The data reporting plan will depend on those who will be using the evaluation 
report. It is often best to assume that a wide range of persons will want to 
read the report. Besides local project personnel and the U.S. Office of 
Education, the range of other persons might include: 
the local board of education; 
school administrators at all levels; 
teachers; 

guidance personnel; 
the state education agency; 
parents and other community residents; and 
other career educacian projects. 

The report should be organised so that it can easily be interpreted by every- 
one who will want access to it. In^^ib^ cases this may mean that special 
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suxntnaries ahovdd be prepared for persons who do not need to know in detail 

the methodology behind the evaluation. A sample format is provided here. 
I, OVERVIEW OF CAREER EDUCATION EFFORT 

The basis for the overview should be the doctiment developed in Task L 

II EVALUATION QUESTIONS ADDRESSED AND SUMMARY OF ANSWERS 

In this part of the report, the evaluation questimis that the study addressed 

shimld stated along with a summary of the findings that relate to the 

questions. 

m. RECOMMENDATIONS FOR PROGRAM MODIFICATIONS 

This part of the report should contain a statement or list of recommenda- 
tions for program change or redirection. In some cases the evaluatlm 
may have raised other areas of interest for study. These areas of 
future interest should be addressed here. 

IV. OUTCOME QUESTION/TREATMENT GROUP MATRIX 

This part of the report should contain a copy of the matrix that was 
developed at the beginning of your evaluation along with a copy of the 
completed Treatment Group-Outcome Area Table. 

V. DISCUSSION OF METHODOLOGY 

This section of the report should include the evaluation design work- 
sheets, copies of locally develof^d instruments, the data collection 
plan and noted deviations, and the final analysis plan. The discussion 
should include any deviation from the original design and any problems 
encountered in carrying out the design. 
VL DETAILED RESULTS 

ERJC This section should include table sfMjd^igu res displaying the data that we re 



collected for the evaluation. 
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APPENDIX 

The appendix should contain copies of the Instruments* If they are not 
Included In VI above. 

The first four sections of this report forn^at can often be used as a summary 
document for distribution to those who do not need the detailed results and 
methodology. 

In developing the outline for the report, attention should be given to actually 
constructing the tables that will be used. After these have been constructed, 
a review of the daU tabulation forms should be made to assure that the 
Information will be available for tabulation In the easiest possible manner. 
By examining these tables at this point, time can be saved In tabulation 
later on. 

Depending on your evaluation design. It may be appropriate to provide 
Interim reports. For Instance, If information Is being collected from 
teachers on a weekly basis regarding what they are doing In their classrooms. 
It may be helpful to report the findings of this activity at various Intervals. 
There are two reasons for this. The Information may be helpful to decision 
mttkers, and the teacher will respond better If he/she receives some 
feedback from his/her efforts. This Is but one example of situations in 
which interim reports may be appropriate. 
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CHAPTER XU 
IMPLEMENTING THE EVALUATION 
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Data Collection 

Following the data collection plan outlined earlier is the key to sound data 
collection efforts. As a part of that plan, a planning chart was developed 
that will provide a task -by -task list of each activity to be carried out. The 
chart provides you with target or completion dates, and indicates the staff 
membes who have prime responsibility and support responsibility for each 
of the tasks. The data collection plan also includes a training needs assess- 
ment and a training plan for the staff involved in the data collection. The 
training should be implemented at least a week before the first data collection 
takes place and alternative arrangements should be made for providing training 
to those who miss the regular training session* In some cases arrangements 
may have to be made to provide separate training for persons with different 
evaluaticm responsibilities who work a different schedule, or are physically 
separated from the rest of the evaluation personnel. 

The types of data to be collected are basically two; these are; 

• activities that occur only one time during the course of the evaluation 
such as pre -test, post»test, certain observations and interviews. 

• activities that are continuous and usually spread over the life of the project* 
An example of one such activity is the regular reports tFiat a teacher might 
submit as a part of the evaluation. These reports are repeated over and 
over on a routine basis* 4 *%m* 
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For the second type of data collection, it is recommended that the evaluation 
staff routinely monitor the data that is being submitted. There are two reasons 
for the monitoring of this information. The first is to assure the quality of 
the data that is being collected. If the quality goes down, corrective measures 
can be taken that will provide a higher quality of data. The second reason is 
to give feedback to the persons coUiecting and submitting the date. The 
feedback is often helpful to the person providing the data and makes the perscm 
more likely to cooperate in all aspects of the evaluation. 

Hopefully, all the tasks related to the collection of data will be completed on 
time and without any difficulty. However, in an evaluation process, various 
events can occur that are outside the influence of the evaluators* The evalua- 
tora must always try to anticipate these events and be flexible enough to 
overcome these obstacles. « The following are some examples of typical 
problems that may occur: 

• A flu epidemic the day of the tests* 

• A lack of cooperation from participating staff or students* 

• Personnel that have ueen closely involved in the evaluation are suddenly 
transferred* 

• New program activities ar^ added that impact on certain outcomes. 

• Budgetary changes are necessary that will reduce evaluation monies* 
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Each of the preceding problems can have a serious e£fect on the evaluation 
study. Of course, there are only a certain number of precautions that one 
can take. For example, by having staff closely involved in the design of the 
evaluation, the chance that staff will not cooperate is minimized. One can- 
not, however, predict the flu. Handling these problems requires the best 
judgment of the staff. In some cases, one might have to modify all or {Mtrt 
of the design. In making any change in the evaluation design, it is important 
to xecord accurately exactly what did occur so that there is a record to refer 
to during the analysis phase* This record of deviations from the original 
design should be included in the evaluation report so that those using 
the information contained in the report understand the context in which the 
findings are to be considered* 

The last step in implementing the data collection plan should be an evaluation 
of the data collection plan and all data collection activities. Often, people 
involved in evaluation make the same errors year after year. Ideally, every 
person involved in the data collection shoulr' "^e involved in tha evaluation. 
If a mmber d the "bugs" in flie data collection plan become apimrent on the day 
that the major collection activity is taking place* these should be noted 
promptly. C^her "bugs" will tarn up when the data are tabulated. An 
evaluation will note any deficiencies and pinpoint or suggest their causes. 
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B* Data Processing And Analysis 

The processing needed to prepare the data for statistical analysis should be 
implemented as soon as possible after the first batch o( data has been collected. 
This would include any necessary labeling for identification purposes^ coding, 
and key -punching, if machine processing is to be used. The process should 
be undertaken early for several reasons: a) the system governing the flow of 
the data can be improved early, b) certain defects in the data can be corrected 
while the fac.j and circumstances are fresh in people^s minds, and c) labeling 
or coding difiiculties can be discovered early, minimizing the need to make 
corrections on large amounts of data which have already been processed. 
Insuring that labeling is error-free is of particular importance in a pre/post 
type of study because of the need to link or pair data from the post-test phase 
with data from the pre-test phase on a student-by ^student and a class^by-class 
basis. This aspect of data processing should be carefully developed or 
reviewed by the computer programmer if machine analysis is to be used, or by 
the statistical consultant if the analysis will be done manually* 

It should be anticipated that the realities of the data collection effort are likely 
to require modifications in the data analysis plans; these would stem chiefly 
from deviations from the original sampling plan and from the fact that incom- 
plete data will have been obtained from various students or various classes. 
Techniques exist for handling or accommodating most of these deviations, and 
it is essential that the analysis specialist be kept informed of such deviaUons 
ati they oc< ur. Vbdifications in the data analysis plan can also be anticipated on 
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the basis of preliminary tabulation o early data; such tabclat on would display 
basic facts about each measured var. vHle, such as means, .i^andard deviations, 
ranges, and sample si^es for each homogeneous grade and treatment group of 
students and classes. Other descriptive statistics may be useful for internal 
program use and/or for refining the data analysis plan; the procedures for 
obtaining these and all other tabulations would be developed by the researcher 
or the statistical consultant. 

It should he pointed out that most studies in education leave insufficient 
time for .arrying out the data analysis. The more time that can be 
allocated for analysis, the greater will be the opportunity to pursue any 
leads, hunches, or questions which each phase of the analysis might 
suggest. The plan, in other words, should be viewed as providing the 
basic framework for analysis, rather than as a rigid prescription. In this 
way, the data will yield the maximum degree and type of meaningful 
information and important cues for interpretation or explanation of the 
basic findings. 

Riporting Findings 

In the broadest sense, the purpose of evaluation is to provide information 
4 oncerning performance to decision makers and planners preparing for the 
future. The evaluation process is not complete until the knowledge gained is 
available to be utilized by planners, managers and policy makers. This, of 
course, is as true for career education programs as it is for others. Because 
career education is a relatively new effort, there does not exist large amounts 
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program information. Therefore, the reporting and dissemination of 
evaluative information is important for individuals seeking to improve 
career education programming at all levels. 

Different kinds of information are needed at different levels of planning and 
administration; the reporting plan should take this into account. As a rule, 
individuals at the operating level (teachers and counselors) require specific 
information on each student (Individual data) while policy makers at the school 
board level need more general information (aggregate data). At the stote and 
federal level the information need becomes increasingly general. Clearly 
policy makers and planners at higher levels do not need^nor could they handle^ 
detailed and specific data from manv sources. The need for increasingly 
general Information at higher policy makit^ levels, carries with it a need to 
make comparisons between programs (treatments), their settings (context) and 
students (participants). The evaluation and reporting plan has been designed to 
provide information which will be useful to all levels of the career education 



By classifying local objectl«res and treatment in terms of Outcome Areas and 
broad activity areas it is possible for planners and policy makers at the federal 
level to compare types of programming and results without being faced with the 
impossible task of trying to make judgments based upon many individual case 
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histories. 
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To assist those not familiar with the context in which the career education 
effort being evaluated occur red.the provision in the report of certain general, 
descriptive nformation would be helpful. Particularly valuable in this regard 
would be information regarding the size of the school district, and the socio- 
economic characteristics of the area. 

In state or federally funded programs, the funding source will usually 
stipulate the kinds of information to be reported. In general, however, in 
reporting findings it is important to describe changes that have occurred in 
the target populations. It is also important to describe the relationships 
between planned change and those changes which actually occurred. In 
reporting change, it is necessary to describe what produced that change. In 
other words, what was different for one student or group of students than for 
others. Or, what was different this year as compared to last year. Without 
this descriptive information it will not be possible to understand the signi- 
ficance of any change that actually occurs. Information that relates to the 
difference between planned and actual change is important in understanding 
which treatments or approaches are most effective, etc. The value of this kind 
of information to planners at all levels should be clear. 

Writing a report without disseminating that report is wasteful. You should 
consider disseminating your results to as many groups as possible. Even if 
your program is not funded by USOE, it would be worthwhile to send them a 
copy. You are strongly urged to send a copy of your final report to the ERIC 
Clearinghouse for Career Education (Northern Illinois University). 
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A large mimber of reports, monograms, and books in the area of evaluation 
of ec..}cational and other public programs was reviewed in the course of 
developing this draft. Aside from general texts cited in the body of the 
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draft, evaluation reports from over 50 career education projects, and 
evaluation guidelines for several federal programs, sources of particular 
note are listed below. The draft plan reflects a syntheses of a multitude 
of ideas contained in these citations. 

Agency for International Development. Evaluation Handbook. Washington, 
D.C. February 1972. 

Anderson, S. B., Ball, S., Murphy, R.T., & Hosethal^ E.J. 
Anatomy of Evaluation: Important Concepts and T^ftiniques in Evaluating 
Education /Training Programs . Educational Testing Service (Sponsored by 
the Office of Naval Research. Princeton, N, J. August, 1973. 

Hatry, Winnie R. , & Fish, D. Practical Program Evaluation for State and 
Local Government Officials. The Urban Institute. Washington, D.C, 1973. 

Kosecoff, J. &Fitz -Gibbon, C. "Many A Slip", Evaluation Comment. 
Center for the Study of Evaluation. December 1973. 

Peck,H. &?Stenner, A.J. Instrumentation and Measurement Strategies for 
Information Based Evaluation (Working Draft). IBEX Inc., Arlington, 
Virginia. 1973. 

Phi Drlta Kappa Study Committee on Evaluation (Daniel Stuff lebeam, et.aU). 
Educational Evaluation and Decision Making . F. E, Peacock Publishers, Inc. 
Itasca, Illinois. 1971. 
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Stenner, k Webster, W.J. Educational Program Audit Handbook . 

Available through: IBEX, Inc., Arlington, Viringia. 1971. 

Stenner, A.J. An O/erview of Information Based Evaluation! A Deaign 
Procedure. IBEX, Inc., Arlington, Virginia. 1972. 

Stenner, A, J. Reporting Evaluation Information? Practical Alternatives 
and Guidelines (Working Draft). IBEX, Inc., Arlington, VirginU. 1973. 

Suchman, Edward A. Evaluative Research! Principles and Practices in 
Public Service and Social Action Programs . Russell Sage Foundation. 
New York. 1967. 

Weiss, Carol H. "Between the Cup and the Lip", Evaluation Concvment . 
Center for the Study of Evaluation. I^cember 1973. 

C. Response Regarding the Self Observation Scales 

The attached comments were received in response to questions raised by the 
instrument review panel from IBEX, Inc. 
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In conversation with National Testing Service, Inc. 
regarding scoring of the SOS, the writer was informed that 
(1) SOS is scored by the "ccmplete factor** nethod with each 
item weighing on each scale; teachers or others would not 
be able to score the SOS, (2) Scoring and reporting services 
are included in the purchase cost of the instrument, 
(3) Any attempt to hand-score would invalidate the factor 
structure^ (4) Con^lete information regarding these areas 
is available in a series of technical papers by the authors 
from National Testing Service, found elsewhere in this document. 

The Achievement Motivation scale is experimental. Whereas 
the other scales were determined by factor analysis. Achievement 
Motivation was determined by discriminant function analysis. 
Work on the validation and replicability of ' Achievement 
Motivation is continuing. 
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Reai>onse Regarding the Differential Aptitude Test, 
The attached table was provided by the Psychological Corporation 
to illustrate grcMip summary data which they are pre(»ared to make 
available to facilitate use of the Differential Aptitude Test In 
program evaluation. 
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Model for Group Summary of Data Available 
for the Differential Aptitude Teat 



In reliitu^n tc* fir^t 

EdtiCdtlcmal 
Aipiration ii: 


Rflatii^n between fipsl 
occupational choice and 
school sul^ect 
fueferencet 


Relation between first 
ocetiDatiOBal choice and 
relevant a{^iitd« 


Distribution oi 


Cases 






Low 


iMo Matcn 


UuaiiiieQ 




XX 


141 Ak 

nlgn 


iMo jviaccii 


Uuaiiii^a 




XX 


Low 


iNO jviatcn 


no Matcn 




XX 


Hign 


A % iff fm # ^ k 

iMO MttlCn 


no Matcn 




XX 




iviaicn 


ivo Matcn 




XX 


Hign 


Matcn 


No Match 


XXX 


XX 


Low 


Matcn 


QuaUfied 


xr 

xxx 


XX 


mgn 


\>ff a^Mt% 

Matcn 


Quauftea 


xxx 


XX 


Low 


Matcn 


Over Qualified 


xxx 


XX 


Hign 


Matcn 


Uver Qualified 


xxx 


VP vr 

XX 


Low 


No Match 


Over Qualified 


xxx 


XX 


Hiffh 






V vv 


VV 
XX 


Match 


Match 


No Match 


xxx 


XX 


Match 


Match 


QuaUfied 


xxx 


XX 


Match 


Match 


Over Qualified 


XXX 


XX 


Match 


No Match 


No Match 


XXX 


XX 


Match 


No Match 


Qualified 


XXX 


XX 


Match , 


No Match 


Over Qualified 


XXX 


XX 



Similar tables would be prepared for the Second and Third Occupational 
Choicec. 



